Attachment 925779 Details for Bug 1128778 – engine log

engine log

engine.log (text/x-log), 3.41 MB, created by Petr Beňas on 2014-08-11 14:23:17 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Petr Beňas

Created: 2014-08-11 14:23:17 UTC

Size: 3.41 MB

patch

obsolete

>2014-07-22 17:03:26,049 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-07-22 17:03:26,050 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-07-22 17:03:26,050 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-07-22 17:03:26,051 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-07-22 17:03:26,052 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-07-22 17:03:26,054 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-07-22 17:03:26,055 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-07-22 17:03:26,065 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-07-22 17:03:26,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-07-22 17:03:26,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-07-22 17:03:26,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-07-22 17:03:26,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-07-22 17:03:26,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-07-22 17:03:26,068 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-07-22 17:03:26,068 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-07-22 17:03:26,071 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-07-22 17:03:26,072 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-07-22 17:03:26,072 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-07-22 17:03:26,072 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432".
>2014-07-22 17:03:26,073 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False".
>2014-07-22 17:03:26,073 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-07-22 17:03:26,074 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-07-22 17:03:26,074 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine".
>2014-07-22 17:03:26,090 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-07-22 17:03:26,090 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-07-22 17:03:26,091 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-07-22 17:03:26,091 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-07-22 17:03:26,092 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-07-22 17:03:26,092 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt".
>2014-07-22 17:03:26,093 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-07-22 17:03:26,093 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-07-22 17:03:26,094 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-07-22 17:03:26,094 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-07-22 17:03:26,103 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-07-22 17:03:26,103 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-07-22 17:03:26,104 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-07-22 17:03:26,104 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-07-22 17:03:26,105 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-07-22 17:03:26,105 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-07-22 17:03:26,106 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-07-22 17:03:26,106 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-07-22 17:03:26,115 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-07-22 17:03:26,116 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-07-22 17:03:26,116 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-07-22 17:03:26,116 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-07-22 17:03:26,117 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-07-22 17:03:26,118 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-07-22 17:03:26,118 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-07-22 17:03:26,126 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-07-22 17:03:26,126 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-07-22 17:03:26,127 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-07-22 17:03:26,128 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-07-22 17:03:26,128 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-07-22 17:03:26,129 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-07-22 17:03:26,129 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-07-22 17:03:26,130 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-07-22 17:03:26,130 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-07-22 17:03:26,130 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10".
>2014-07-22 17:03:26,136 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-07-22 17:03:26,136 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-07-22 17:03:26,137 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-07-22 17:03:26,137 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt".
>2014-07-22 17:03:26,138 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-07-22 17:03:26,138 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-07-22 17:03:26,139 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-07-22 17:03:26,139 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-07-22 17:03:26,146 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-07-22 17:03:26,146 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-07-22 17:03:26,950 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend
>2014-07-22 17:03:27,568 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-07-22 17:03:27,691 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140715172116.git4687dc1.el6
>2014-07-22 17:03:27,692 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries
>2014-07-22 17:03:27,705 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries
>2014-07-22 17:03:27,706 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager
>2014-07-22 17:03:27,707 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-07-22 17:03:27,748 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager
>2014-07-22 17:03:27,748 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager
>2014-07-22 17:03:27,749 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-07-22 17:03:27,750 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager
>2014-07-22 17:03:27,753 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector
>2014-07-22 17:03:27,780 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree
>2014-07-22 17:03:27,797 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector
>2014-07-22 17:03:27,800 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer
>2014-07-22 17:03:27,815 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer
>2014-07-22 17:03:27,971 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers
>2014-07-22 17:03:28,003 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-07-22 17:03:28,023 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-07-22 17:03:28,024 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-07-22 17:03:28,025 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-07-22 17:03:28,026 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-07-22 17:03:28,026 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN
>2014-07-22 17:03:28,071 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager
>2014-07-22 17:03:28,072 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager
>2014-07-22 17:03:28,073 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager
>2014-07-22 17:03:28,076 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager
>2014-07-22 17:03:28,077 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector
>2014-07-22 17:03:28,081 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector
>2014-07-22 17:03:28,143 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list
>2014-07-22 17:03:28,147 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-07-22 17:03:28,148 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-07-22 17:03:28,148 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list
>2014-07-22 17:03:28,257 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager
>2014-07-22 17:03:28,264 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager
>2014-07-22 17:03:28,288 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully.
>2014-07-22 17:03:28,296 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager
>2014-07-22 17:03:28,318 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) The user admin from authz extension internal got updated since last interval
>2014-07-22 17:03:28,418 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager
>2014-07-22 17:03:28,424 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully.
>2014-07-22 17:03:28,424 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer
>2014-07-22 17:03:28,425 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer
>2014-07-22 17:03:28,425 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check
>2014-07-22 17:03:28,428 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check
>2014-07-22 17:03:28,438 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-07-22 17:03:28,438 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities
>2014-07-22 17:03:28,440 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities
>2014-07-22 17:03:28,445 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager
>2014-07-22 17:03:28,476 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255
>2014-07-22 17:03:28,521 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped
>2014-07-22 17:03:28,521 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager
>2014-07-22 17:03:28,522 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat
>2014-07-22 17:03:28,527 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized
>2014-07-22 17:03:59,478 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-07-22 17:03:59,487 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-07-22 17:03:59,494 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: UNASSIGNED not exist in string table
>2014-07-22 17:03:59,495 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-07-22 17:03:59,496 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-07-22 17:03:59,498 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-07-22 17:03:59,499 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-07-22 17:03:59,500 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-07-22 17:03:59,501 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-07-22 17:03:59,503 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-07-22 17:03:59,504 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-07-22 17:03:59,505 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-07-22 17:03:59,513 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_STOPPED not exist in string table
>2014-07-22 17:03:59,514 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_STARTED not exist in string table
>2014-07-22 17:03:59,515 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_ERROR not exist in string table
>2014-07-22 17:03:59,516 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-07-22 17:03:59,517 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-07-22 17:03:59,518 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-07-22 17:03:59,519 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-07-22 17:03:59,798 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-22 17:03:59,844 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [429a523e] Running command: LogoutBySessionCommand internal: false.
>2014-07-22 17:03:59,864 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [2a696afa] Running command: LogoutUserCommand internal: false.
>2014-07-22 17:03:59,883 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [2a696afa] Correlation ID: 2a696afa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-22 17:08:18,598 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-22 17:08:18,609 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-22 17:08:18,625 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [6daa0b72] Running command: LogoutBySessionCommand internal: false.
>2014-07-22 17:08:18,637 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [1c8ad795] Running command: LogoutUserCommand internal: false.
>2014-07-22 17:08:18,651 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1c8ad795] Correlation ID: 1c8ad795, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-22 17:11:27,720 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-22 17:11:27,731 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-22 17:11:27,741 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [56e5ba51] Running command: LogoutBySessionCommand internal: false.
>2014-07-22 17:11:27,747 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [4cd9ad06] Running command: LogoutUserCommand internal: false.
>2014-07-22 17:11:27,753 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4cd9ad06] Correlation ID: 4cd9ad06, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-22 17:14:17,831 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-07-22 17:14:17,836 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-22 17:14:17,849 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [66ac6b48] Running command: LogoutBySessionCommand internal: false.
>2014-07-22 17:14:17,861 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [51e93529] Running command: LogoutUserCommand internal: false.
>2014-07-22 17:14:17,868 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [51e93529] Correlation ID: 51e93529, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-22 18:03:28,330 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) The user admin from authz extension internal got updated since last interval
>2014-07-22 18:03:28,423 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf.
>2014-07-22 19:03:28,340 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) The user admin from authz extension internal got updated since last interval
>2014-07-22 19:03:28,431 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf.
>2014-07-22 20:03:28,348 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) The user admin from authz extension internal got updated since last interval
>2014-07-22 20:03:28,434 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf.
>2014-07-22 21:03:28,355 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) The user admin from authz extension internal got updated since last interval
>2014-07-22 21:03:28,437 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf.
>2014-07-22 22:03:28,365 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) The user admin from authz extension internal got updated since last interval
>2014-07-22 22:03:28,442 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf.
>2014-07-22 23:03:28,375 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) The user admin from authz extension internal got updated since last interval
>2014-07-22 23:03:28,445 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf.
>2014-07-23 00:03:28,382 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) The user admin from authz extension internal got updated since last interval
>2014-07-23 00:03:28,448 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf.
>2014-07-23 01:03:28,389 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) The user admin from authz extension internal got updated since last interval
>2014-07-23 01:03:28,449 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf.
>2014-07-23 02:03:28,395 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) The user admin from authz extension internal got updated since last interval
>2014-07-23 02:03:28,452 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-23 03:03:28,402 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) The user admin from authz extension internal got updated since last interval
>2014-07-23 03:03:28,455 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
>2014-07-23 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-18) Start deleteAgedOutCommandEntities
>2014-07-23 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Start deleteAgedOutAuditLogs
>2014-07-23 03:35:35,015 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-18) Finished deleteAgedOutCommandEntities
>2014-07-23 03:35:35,019 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Finished deleteAgedOutAuditLogs
>2014-07-23 04:03:28,411 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) The user admin from authz extension internal got updated since last interval
>2014-07-23 04:03:28,456 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf.
>2014-07-23 05:03:28,417 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) The user admin from authz extension internal got updated since last interval
>2014-07-23 05:03:28,458 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf.
>2014-07-23 06:03:28,422 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval
>2014-07-23 06:03:28,460 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-07-23 07:03:28,428 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) The user admin from authz extension internal got updated since last interval
>2014-07-23 07:03:28,463 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-07-23 08:03:28,437 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) The user admin from authz extension internal got updated since last interval
>2014-07-23 08:03:28,464 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf.
>2014-07-23 09:03:28,444 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) The user admin from authz extension internal got updated since last interval
>2014-07-23 09:03:28,468 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf.
>2014-07-23 10:03:28,455 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) The user admin from authz extension internal got updated since last interval
>2014-07-23 10:03:28,471 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf.
>2014-07-23 11:03:28,466 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) The user admin from authz extension internal got updated since last interval
>2014-07-23 11:03:28,473 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
>2014-07-23 12:03:28,475 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-07-23 12:03:28,481 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) The user admin from authz extension internal got updated since last interval
>2014-07-23 13:03:28,477 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf.
>2014-07-23 13:03:28,491 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) The user admin from authz extension internal got updated since last interval
>2014-07-23 14:03:28,481 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf.
>2014-07-23 14:03:28,497 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) The user admin from authz extension internal got updated since last interval
>2014-07-23 15:03:28,483 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf.
>2014-07-23 15:03:28,502 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) The user admin from authz extension internal got updated since last interval
>2014-07-23 16:03:28,485 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf.
>2014-07-23 16:03:28,512 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) The user admin from authz extension internal got updated since last interval
>2014-07-23 17:03:28,488 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf.
>2014-07-23 17:03:28,523 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-16) The user admin from authz extension internal got updated since last interval
>2014-07-23 18:03:28,490 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-07-23 18:03:28,535 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) The user admin from authz extension internal got updated since last interval
>2014-07-23 19:03:28,493 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-07-23 19:03:28,542 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval
>2014-07-23 20:03:28,495 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-07-23 20:03:28,549 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) The user admin from authz extension internal got updated since last interval
>2014-07-23 21:03:28,498 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf.
>2014-07-23 21:03:28,559 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) The user admin from authz extension internal got updated since last interval
>2014-07-23 22:03:28,500 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf.
>2014-07-23 22:03:28,568 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-57) The user admin from authz extension internal got updated since last interval
>2014-07-23 23:03:28,503 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
>2014-07-23 23:03:28,574 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) The user admin from authz extension internal got updated since last interval
>2014-07-24 00:03:28,506 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf.
>2014-07-24 00:03:28,583 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) The user admin from authz extension internal got updated since last interval
>2014-07-24 01:03:28,508 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
>2014-07-24 01:03:28,589 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) The user admin from authz extension internal got updated since last interval
>2014-07-24 02:03:28,512 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf.
>2014-07-24 02:03:28,594 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) The user admin from authz extension internal got updated since last interval
>2014-07-24 03:03:28,515 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf.
>2014-07-24 03:03:28,601 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) The user admin from authz extension internal got updated since last interval
>2014-07-24 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) Start deleteAgedOutAuditLogs
>2014-07-24 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Start deleteAgedOutCommandEntities
>2014-07-24 03:35:35,007 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) Finished deleteAgedOutAuditLogs
>2014-07-24 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Finished deleteAgedOutCommandEntities
>2014-07-24 04:03:28,517 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
>2014-07-24 04:03:28,608 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) The user admin from authz extension internal got updated since last interval
>2014-07-24 05:03:28,520 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf.
>2014-07-24 05:03:28,613 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) The user admin from authz extension internal got updated since last interval
>2014-07-24 06:03:28,522 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf.
>2014-07-24 06:03:28,618 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) The user admin from authz extension internal got updated since last interval
>2014-07-24 07:03:28,524 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf.
>2014-07-24 07:03:28,626 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) The user admin from authz extension internal got updated since last interval
>2014-07-24 08:03:28,527 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
>2014-07-24 08:03:28,633 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) The user admin from authz extension internal got updated since last interval
>2014-07-24 09:03:28,529 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf.
>2014-07-24 09:03:28,639 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) The user admin from authz extension internal got updated since last interval
>2014-07-24 09:34:22,185 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false.
>2014-07-24 09:34:22,202 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-24 09:34:24,208 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-24 09:40:48,018 INFO  [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-5) [1270305f] Running command: AddVdsCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-24 09:40:48,147 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:40:48,166 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: REGISTER_VDS
>, sharedLocks= ]
>2014-07-24 09:40:48,175 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 25de2f7b
>2014-07-24 09:40:48,177 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] FINISH, RemoveVdsVDSCommand, log id: 25de2f7b
>2014-07-24 09:40:48,180 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 75f74996
>2014-07-24 09:40:48,182 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b
>2014-07-24 09:40:48,185 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager
>2014-07-24 09:40:48,191 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-5) [680a0cb6] Entered VdsManager constructor
>2014-07-24 09:40:48,214 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-5) [680a0cb6] Initialize vdsBroker (10.34.62.208,54,321)
>2014-07-24 09:40:48,368 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-5) [680a0cb6] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-07-24 09:40:48,377 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] FINISH, AddVdsVDSCommand, log id: 75f74996
>2014-07-24 09:40:48,394 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [680a0cb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-24 09:40:48,448 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-24 09:40:48,453 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [680a0cb6] Correlation ID: 1270305f, Job ID: 06b707e0-e985-4e05-a58a-ddecf62d1c41, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin.
>2014-07-24 09:40:48,475 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Running command: InstallVdsInternalCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:40:48,488 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Before Installation host 17f7fe56-5f32-40ef-b098-611b84484e0b, rack
>2014-07-24 09:40:48,504 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1388ca8d
>2014-07-24 09:40:48,533 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] FINISH, SetVdsStatusVDSCommand, log id: 1388ca8d
>2014-07-24 09:40:48,595 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Installation 10.34.62.208: Connected to host 10.34.62.208 with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff
>2014-07-24 09:40:48,652 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host 10.34.62.208 with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff.
>2014-07-24 09:40:48,672 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Installation of 10.34.62.208. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
>2014-07-24 09:40:48,676 INFO  [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-8-thread-8) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating
>2014-07-24 09:40:48,807 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-8) SSH execute root@10.34.62.208 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True'
>2014-07-24 09:40:49,915 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Initializing
>2014-07-24 09:40:49,923 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing.
>2014-07-24 09:40:49,937 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment setup
>2014-07-24 09:40:49,945 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup.
>2014-07-24 09:40:49,955 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment packages setup
>2014-07-24 09:40:49,962 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup.
>2014-07-24 09:40:51,371 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-07-24 09:41:24,336 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Downloading: ovirt-3.5-epel/metalink (0%)
>2014-07-24 09:41:24,344 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Downloading: ovirt-3.5-epel/metalink (0%).
>2014-07-24 09:41:32,025 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Programs detection
>2014-07-24 09:41:32,032 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection.
>2014-07-24 09:41:32,033 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment customization
>2014-07-24 09:41:32,038 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization.
>2014-07-24 09:42:02,467 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Kdump supported
>2014-07-24 09:42:02,475 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported.
>2014-07-24 09:42:02,498 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Logs at host located at: '/tmp/ovirt-host-deploy-20140724094049-nekf10.log'
>2014-07-24 09:42:02,505 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140724094049-nekf10.log'.
>2014-07-24 09:42:02,598 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.34.62.208 reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A
>2014-07-24 09:42:02,629 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host 10.34.62.208
>2014-07-24 09:42:02,713 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Setup validation
>2014-07-24 09:42:02,719 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation.
>2014-07-24 09:42:02,755 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Hardware supports virtualization
>2014-07-24 09:42:02,761 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization.
>2014-07-24 09:42:03,007 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Transaction setup
>2014-07-24 09:42:03,016 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup.
>2014-07-24 09:42:03,061 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Misc configuration
>2014-07-24 09:42:03,070 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration.
>2014-07-24 09:42:03,071 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Package installation
>2014-07-24 09:42:03,078 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation.
>2014-07-24 09:42:44,942 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Downloading Packages
>2014-07-24 09:42:44,952 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Downloading Packages.
>2014-07-24 09:42:44,968 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: ioprocess-0.5.0-1.el6.x86_64
>2014-07-24 09:42:44,975 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ioprocess-0.5.0-1.el6.x86_64.
>2014-07-24 09:42:46,340 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: kexec-tools-2.0.0-273.1.el6.x86_64
>2014-07-24 09:42:46,347 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: kexec-tools-2.0.0-273.1.el6.x86_64.
>2014-07-24 09:42:46,349 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: mom-0.4.1-2.el6.noarch
>2014-07-24 09:42:46,355 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: mom-0.4.1-2.el6.noarch.
>2014-07-24 09:42:46,366 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: python-ioprocess-0.5.0-1.el6.noarch
>2014-07-24 09:42:46,373 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-ioprocess-0.5.0-1.el6.noarch.
>2014-07-24 09:42:47,996 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64
>2014-07-24 09:42:48,004 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64.
>2014-07-24 09:42:49,776 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-4.16.0-42.git3bfad86.el6.x86_64
>2014-07-24 09:42:49,787 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-4.16.0-42.git3bfad86.el6.x86_64.
>2014-07-24 09:42:49,989 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:49,998 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:50,408 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:50,417 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:50,789 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64
>2014-07-24 09:42:50,796 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64.
>2014-07-24 09:42:50,977 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:50,984 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:51,197 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:51,210 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:51,381 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:51,387 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:51,506 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Check Package Signatures
>2014-07-24 09:42:51,515 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Check Package Signatures.
>2014-07-24 09:42:51,523 WARN  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7
>2014-07-24 09:42:51,532 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Host rack installation in progress . Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7.
>2014-07-24 09:42:51,664 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Running Test Transaction
>2014-07-24 09:42:51,673 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Test Transaction.
>2014-07-24 09:42:51,963 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Running Transaction
>2014-07-24 09:42:51,970 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Transaction.
>2014-07-24 09:42:52,404 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 1/22: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:52,410 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 1/22: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:52,691 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum install: 2/22: ioprocess-0.5.0-1.el6.x86_64
>2014-07-24 09:42:52,698 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 2/22: ioprocess-0.5.0-1.el6.x86_64.
>2014-07-24 09:42:53,068 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum install: 3/22: python-ioprocess-0.5.0-1.el6.noarch
>2014-07-24 09:42:53,078 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 3/22: python-ioprocess-0.5.0-1.el6.noarch.
>2014-07-24 09:42:53,390 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 4/22: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64
>2014-07-24 09:42:53,397 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 4/22: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64.
>2014-07-24 09:42:53,907 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 5/22: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:53,913 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 5/22: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:54,207 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 6/22: mom-0.4.1-2.el6.noarch
>2014-07-24 09:42:54,212 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 6/22: mom-0.4.1-2.el6.noarch.
>2014-07-24 09:42:54,799 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 7/22: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:54,807 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 7/22: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:55,074 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 8/22: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:42:55,080 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 8/22: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:42:55,514 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 9/22: vdsm-4.16.0-42.git3bfad86.el6.x86_64
>2014-07-24 09:42:55,521 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 9/22: vdsm-4.16.0-42.git3bfad86.el6.x86_64.
>2014-07-24 09:43:04,446 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 10/22: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch
>2014-07-24 09:43:04,452 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 10/22: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch.
>2014-07-24 09:43:04,791 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 11/22: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64
>2014-07-24 09:43:04,798 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 11/22: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64.
>2014-07-24 09:43:05,135 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 12/22: kexec-tools-2.0.0-273.1.el6.x86_64
>2014-07-24 09:43:05,143 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 12/22: kexec-tools-2.0.0-273.1.el6.x86_64.
>2014-07-24 09:43:19,524 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 13/22: vdsm-cli
>2014-07-24 09:43:19,532 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 13/22: vdsm-cli.
>2014-07-24 09:43:19,831 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 14/22: vdsm
>2014-07-24 09:43:19,837 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 14/22: vdsm.
>2014-07-24 09:43:20,985 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 15/22: vdsm-jsonrpc
>2014-07-24 09:43:20,993 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 15/22: vdsm-jsonrpc.
>2014-07-24 09:43:21,304 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 16/22: vdsm-xmlrpc
>2014-07-24 09:43:21,310 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 16/22: vdsm-xmlrpc.
>2014-07-24 09:43:21,492 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 17/22: vdsm-python
>2014-07-24 09:43:21,499 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 17/22: vdsm-python.
>2014-07-24 09:43:21,820 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 18/22: vdsm-python-zombiereaper
>2014-07-24 09:43:21,838 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 18/22: vdsm-python-zombiereaper.
>2014-07-24 09:43:22,015 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 19/22: vdsm-yajsonrpc
>2014-07-24 09:43:22,022 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 19/22: vdsm-yajsonrpc.
>2014-07-24 09:43:22,236 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 20/22: mom
>2014-07-24 09:43:22,242 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 20/22: mom.
>2014-07-24 09:43:22,616 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 21/22: qemu-kvm-rhev-tools
>2014-07-24 09:43:22,622 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 21/22: qemu-kvm-rhev-tools.
>2014-07-24 09:43:22,858 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 22/22: kexec-tools
>2014-07-24 09:43:22,864 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 22/22: kexec-tools.
>2014-07-24 09:43:27,395 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 1/22: vdsm-python-zombiereaper.noarch 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:27,402 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 1/22: vdsm-python-zombiereaper.noarch 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:27,497 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 2/22: python-ioprocess.noarch 0:0.5.0-1.el6 - u
>2014-07-24 09:43:27,505 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 2/22: python-ioprocess.noarch 0:0.5.0-1.el6 - u.
>2014-07-24 09:43:27,655 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 3/22: vdsm-python.x86_64 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:27,663 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 3/22: vdsm-python.x86_64 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:27,665 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 4/22: vdsm-cli.noarch 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:27,672 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 4/22: vdsm-cli.noarch 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:27,716 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 5/22: vdsm.x86_64 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:27,722 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 5/22: vdsm.x86_64 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:27,816 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 6/22: kexec-tools.x86_64 0:2.0.0-273.1.el6 - u
>2014-07-24 09:43:27,822 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 6/22: kexec-tools.x86_64 0:2.0.0-273.1.el6 - u.
>2014-07-24 09:43:27,901 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 7/22: vdsm-jsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:27,909 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 7/22: vdsm-jsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:27,970 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 8/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.12 - u
>2014-07-24 09:43:27,977 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 8/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.12 - u.
>2014-07-24 09:43:28,044 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 9/22: vdsm-yajsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:28,052 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 9/22: vdsm-yajsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:28,111 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 10/22: vdsm-xmlrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u
>2014-07-24 09:43:28,119 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 10/22: vdsm-xmlrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u.
>2014-07-24 09:43:28,179 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 11/22: mom.noarch 0:0.4.1-2.el6 - u
>2014-07-24 09:43:28,188 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 11/22: mom.noarch 0:0.4.1-2.el6 - u.
>2014-07-24 09:43:28,256 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 12/22: ioprocess.x86_64 0:0.5.0-1.el6 - u
>2014-07-24 09:43:28,264 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 12/22: ioprocess.x86_64 0:0.5.0-1.el6 - u.
>2014-07-24 09:43:28,304 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 13/22: vdsm-yajsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,313 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 13/22: vdsm-yajsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,315 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 14/22: kexec-tools.x86_64 0:2.0.0-273.el6 - ud
>2014-07-24 09:43:28,321 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 14/22: kexec-tools.x86_64 0:2.0.0-273.el6 - ud.
>2014-07-24 09:43:28,322 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 15/22: mom.noarch 0:0.4.0-1.el6ev - ud
>2014-07-24 09:43:28,328 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 15/22: mom.noarch 0:0.4.0-1.el6ev - ud.
>2014-07-24 09:43:28,329 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 16/22: vdsm-jsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,335 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 16/22: vdsm-jsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,337 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 17/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.10 - ud
>2014-07-24 09:43:28,343 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 17/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.10 - ud.
>2014-07-24 09:43:28,344 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 18/22: vdsm-cli.noarch 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,352 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 18/22: vdsm-cli.noarch 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,353 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 19/22: vdsm-python.x86_64 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,359 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 19/22: vdsm-python.x86_64 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,360 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 20/22: vdsm.x86_64 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,368 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 20/22: vdsm.x86_64 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,370 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 21/22: vdsm-python-zombiereaper.noarch 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,375 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 21/22: vdsm-python-zombiereaper.noarch 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,377 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 22/22: vdsm-xmlrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud
>2014-07-24 09:43:28,382 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 22/22: vdsm-xmlrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud.
>2014-07-24 09:43:28,440 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Misc configuration
>2014-07-24 09:43:28,447 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration.
>2014-07-24 09:43:28,530 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Setting time
>2014-07-24 09:43:28,536 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time.
>2014-07-24 09:43:30,148 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Enrolling certificate
>2014-07-24 09:43:30,154 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate.
>2014-07-24 09:43:31,498 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Transaction commit
>2014-07-24 09:43:31,505 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit.
>2014-07-24 09:43:31,634 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Closing up
>2014-07-24 09:43:31,641 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up.
>2014-07-24 09:43:32,051 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Restarting kdump
>2014-07-24 09:43:32,058 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Restarting kdump.
>2014-07-24 09:43:45,267 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stopping libvirtd
>2014-07-24 09:43:45,273 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd.
>2014-07-24 09:43:45,534 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Starting vdsm
>2014-07-24 09:43:45,540 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm.
>2014-07-24 09:43:47,326 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Pre-termination
>2014-07-24 09:43:47,338 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination.
>2014-07-24 09:43:47,349 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140724094347-10.34.62.208-43a66d5e.log'
>2014-07-24 09:43:47,358 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140724094347-10.34.62.208-43a66d5e.log'.
>2014-07-24 09:43:47,624 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Termination
>2014-07-24 09:43:47,631 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination.
>2014-07-24 09:43:48,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-07-24 09:43:48,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Command PollVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-24 09:43:48,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-07-24 09:43:48,673 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Command PollVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-24 09:43:49,196 INFO  [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Engine managed to communicate with VDSM agent on host rack
>2014-07-24 09:43:49,632 INFO  [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] The management network ovirtmgmt is already configured on host rack
>2014-07-24 09:43:49,635 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 596b9dcb
>2014-07-24 09:43:49,642 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] FINISH, SetVdsStatusVDSCommand, log id: 596b9dcb
>2014-07-24 09:43:49,643 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] After Installation host rack, VDS
>2014-07-24 09:43:49,659 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Correlation ID: 43a66d5e, Job ID: 06b707e0-e985-4e05-a58a-ddecf62d1c41, Call Stack: null, Custom Event ID: -1, Message: Host rack installed
>2014-07-24 09:43:49,676 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-24 09:43:51,819 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 30cf88c4
>2014-07-24 09:43:51,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GetHardwareInfoVDSCommand, log id: 30cf88c4
>2014-07-24 09:43:52,016 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [6386d02e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:43:52,141 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (DefaultQuartzScheduler_Worker-50) [6aa9f65f] Running command: UpdateVdsGroupCommand internal: true. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-24 09:43:52,176 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [6aa9f65f] Correlation ID: 6aa9f65f, Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was updated by system
>2014-07-24 09:43:52,221 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:43:52,225 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=NonOperational, nonOperationalReason=RNG_SOURCES_INCOMPATIBLE_WITH_CLUSTER, stopSpmFailureLogged=false), log id: 70eeacec
>2014-07-24 09:43:52,232 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] FINISH, SetVdsStatusVDSCommand, log id: 70eeacec
>2014-07-24 09:43:52,258 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1c293098] Correlation ID: 1c293098, Job ID: a3e8abcc-1329-43dd-b37b-ecbb43a04a3a, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default Random Number Generator sources. The Hosts supported sources are: Non interactive user; and the cluster requirements are: RANDOM.
>2014-07-24 09:43:52,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2f58d973
>2014-07-24 09:43:52,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] FINISH, GetHardwareInfoVDSCommand, log id: 2f58d973
>2014-07-24 09:43:52,406 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:43:52,421 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational.
>2014-07-24 09:43:52,462 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[a2660663-ef34-41ee-a580-0fc425183c36]), log id: 2472108a
>2014-07-24 09:43:52,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] FINISH, FullListVdsCommand, return: [{displaySecurePort=-1, kvmEnable=true, nicModel=rtl8139,pv, guestIPs=, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, username=, copyPasteEnable=true, timeOffset=7200, transparentHugePages=true, vmId=a2660663-ef34-41ee-a580-0fc425183c36, acpiEnable=true, custom={device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7device_62de13f5-3461-4945-9cec-d2cc141a05d0device_7ee4ae57-89f0-4b38-8f28-fc4a3ed37f7d=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=7ee4ae57-89f0-4b38-8f28-fc4a3ed37f7d, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x05, bus=0x00, domain=0x0000, type=pci, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7device_62de13f5-3461-4945-9cec-d2cc141a05d0=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=62de13f5-3461-4945-9cec-d2cc141a05d0, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4b=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=16e5639c-8110-45fa-b60d-ab8598d0ed4b, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=3}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=3f51e57f-378c-4c48-9190-8893bc06a9a8, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=43b633f0-4db0-4d8f-b872-fb2f6b5657a7, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=0a15b13b-427e-4147-9179-0dcdc4094307, device=memballoon, type=BALLOON, bootOrder=0, specParams={model=virtio}, address={slot=0x07, bus=0x00, domain=0x0000, type=pci, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=balloon0, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=5192, displayPort=-1, smp=1, status=Down, emulatedMachine=rhel6.5.0, vmType=kvm, memGuaranteedSize=1024, display=qxl, exitCode=0, pid=10456, exitReason=7, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, guestFQDN=, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, exitMessage=User shut down from within the guest, clientIp=, devices=[Ljava.lang.Object;@7f860606, vmName=ffffuuu, fileTransferEnable=true, cpuType=Westmere}], log id: 2472108a
>2014-07-24 09:43:52,503 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Importing VM ffffuuu as external-ffffuuu, as it is running on the on Host, but does not exist in the engine.
>2014-07-24 09:43:52,892 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Lock Acquired to object EngineLock [exclusiveLocks= key: external-ffffuuu value: VM_NAME
>, sharedLocks= ]
>2014-07-24 09:43:52,972 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Running command: AddVmFromScratchCommand internal: true. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-24 09:43:53,219 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [7a179a08] Correlation ID: 7a179a08, Call Stack: null, Custom Event ID: -1, Message: VM external-ffffuuu was created by Non interactive user.
>2014-07-24 09:43:53,221 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Lock freed to object EngineLock [exclusiveLocks= key: external-ffffuuu value: VM_NAME
>, sharedLocks= ]
>2014-07-24 09:43:53,247 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [43ad7acf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:43:53,254 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [17b882b7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 09:43:53,257 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [17b882b7] Host 17f7fe56-5f32-40ef-b098-611b84484e0b : rack is already in NonOperational status for reason RNG_SOURCES_INCOMPATIBLE_WITH_CLUSTER. SetNonOperationalVds command is skipped.
>2014-07-24 10:03:28,532 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-24 10:03:28,644 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) The user admin from authz extension internal got updated since last interval
>2014-07-24 10:27:46,095 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp--127.0.0.1-8702-7) [2be7a4c0] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-24 10:27:46,109 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2be7a4c0] Correlation ID: 2be7a4c0, Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was updated by admin
>2014-07-24 10:27:51,309 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-1) [3659b7c9] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-24 10:27:51,346 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 10:27:51,348 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Before acquiring lock in order to prevent monitoring for host rack from data-center Default
>2014-07-24 10:27:51,349 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default
>2014-07-24 10:27:51,359 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5bfe183e
>2014-07-24 10:27:51,370 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] FINISH, SetVdsStatusVDSCommand, log id: 5bfe183e
>2014-07-24 10:27:51,431 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default
>2014-07-24 10:27:51,444 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Correlation ID: 3659b7c9, Job ID: 5d9d183f-1589-40e0-9849-9989f271eb87, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin.
>2014-07-24 10:27:51,450 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-24 10:27:53,006 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6e79d3fb
>2014-07-24 10:27:53,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, GetHardwareInfoVDSCommand, log id: 6e79d3fb
>2014-07-24 10:27:53,108 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 10:27:53,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=a2660663-ef34-41ee-a580-0fc425183c36, force=false, secondsToWait=0, gracefully=false, reason=), log id: 22cc1601
>2014-07-24 10:27:53,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] FINISH, DestroyVDSCommand, log id: 22cc1601
>2014-07-24 10:27:53,287 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-24 10:27:53,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f3e13e1
>2014-07-24 10:27:53,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7f3e13e1
>2014-07-24 10:27:53,495 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-24 10:27:53,501 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-24 10:27:53,544 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [4ab804d6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-24 11:03:28,535 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-07-24 11:03:28,654 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) The user admin from authz extension internal got updated since last interval
>2014-07-24 12:03:28,538 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf.
>2014-07-24 12:03:28,662 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) The user admin from authz extension internal got updated since last interval
>2014-07-24 13:03:28,542 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-07-24 13:03:28,669 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) The user admin from authz extension internal got updated since last interval
>2014-07-24 14:03:28,545 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf.
>2014-07-24 14:03:28,675 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) The user admin from authz extension internal got updated since last interval
>2014-07-24 15:03:28,548 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-07-24 15:03:28,681 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval
>2014-07-24 15:53:59,509 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-24 15:53:59,515 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-24 15:53:59,530 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [19f1d247] Running command: LogoutBySessionCommand internal: false.
>2014-07-24 15:53:59,534 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [43730762] Running command: LogoutUserCommand internal: false.
>2014-07-24 15:53:59,542 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [43730762] Correlation ID: 43730762, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-24 16:03:28,551 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-24 16:03:28,690 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) The user admin from authz extension internal got updated since last interval
>2014-07-24 17:03:28,554 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf.
>2014-07-24 17:03:28,696 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval
>2014-07-24 18:03:28,557 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-07-24 18:03:28,704 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) The user admin from authz extension internal got updated since last interval
>2014-07-24 19:03:28,560 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-07-24 19:03:28,717 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval
>2014-07-24 20:03:28,562 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-24 20:03:28,724 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) The user admin from authz extension internal got updated since last interval
>2014-07-24 21:03:28,565 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf.
>2014-07-24 21:03:28,731 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval
>2014-07-24 22:03:28,568 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf.
>2014-07-24 22:03:28,738 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) The user admin from authz extension internal got updated since last interval
>2014-07-24 23:03:28,570 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf.
>2014-07-24 23:03:28,744 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval
>2014-07-25 00:03:28,572 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-07-25 00:03:28,752 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) The user admin from authz extension internal got updated since last interval
>2014-07-25 01:03:28,578 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf.
>2014-07-25 01:03:28,759 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) The user admin from authz extension internal got updated since last interval
>2014-07-25 02:03:28,581 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
>2014-07-25 02:03:28,764 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) The user admin from authz extension internal got updated since last interval
>2014-07-25 03:03:28,583 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf.
>2014-07-25 03:03:28,770 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) The user admin from authz extension internal got updated since last interval
>2014-07-25 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Start deleteAgedOutAuditLogs
>2014-07-25 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-72) Start deleteAgedOutCommandEntities
>2014-07-25 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Finished deleteAgedOutAuditLogs
>2014-07-25 03:35:35,008 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-72) Finished deleteAgedOutCommandEntities
>2014-07-25 04:03:28,586 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-07-25 04:03:28,782 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) The user admin from authz extension internal got updated since last interval
>2014-07-25 05:03:28,589 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-07-25 05:03:28,790 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) The user admin from authz extension internal got updated since last interval
>2014-07-25 06:03:28,592 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
>2014-07-25 06:03:28,798 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) The user admin from authz extension internal got updated since last interval
>2014-07-25 07:03:28,595 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-07-25 07:03:28,808 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) The user admin from authz extension internal got updated since last interval
>2014-07-25 08:03:28,597 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
>2014-07-25 08:03:28,815 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval
>2014-07-25 09:03:28,599 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
>2014-07-25 09:03:28,823 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) The user admin from authz extension internal got updated since last interval
>2014-07-25 10:03:28,601 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
>2014-07-25 10:03:28,832 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval
>2014-07-25 11:03:28,604 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-07-25 11:03:28,841 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) The user admin from authz extension internal got updated since last interval
>2014-07-25 12:03:28,606 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf.
>2014-07-25 12:03:28,847 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) The user admin from authz extension internal got updated since last interval
>2014-07-25 12:23:52,535 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-07-25 12:23:52,536 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-07-25 12:23:52,537 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-07-25 12:23:52,537 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-07-25 12:23:52,552 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-07-25 12:23:52,553 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-07-25 12:23:52,553 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-07-25 12:23:52,553 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-07-25 12:23:52,554 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-07-25 12:23:52,554 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-07-25 12:23:52,555 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-07-25 12:23:52,555 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-07-25 12:23:52,556 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-07-25 12:23:52,556 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-07-25 12:23:52,563 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-07-25 12:23:52,563 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-07-25 12:23:52,563 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-07-25 12:23:52,564 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-07-25 12:23:52,564 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432".
>2014-07-25 12:23:52,565 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False".
>2014-07-25 12:23:52,565 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-07-25 12:23:52,567 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-07-25 12:23:52,567 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine".
>2014-07-25 12:23:52,568 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-07-25 12:23:52,568 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-07-25 12:23:52,570 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-07-25 12:23:52,571 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-07-25 12:23:52,572 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-07-25 12:23:52,572 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt".
>2014-07-25 12:23:52,572 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-07-25 12:23:52,573 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-07-25 12:23:52,573 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-07-25 12:23:52,574 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-07-25 12:23:52,580 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-07-25 12:23:52,581 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-07-25 12:23:52,581 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-07-25 12:23:52,584 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-07-25 12:23:52,584 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-07-25 12:23:52,585 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-07-25 12:23:52,585 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-07-25 12:23:52,586 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-07-25 12:23:52,586 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-07-25 12:23:52,587 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-07-25 12:23:52,587 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-07-25 12:23:52,587 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-07-25 12:23:52,588 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-07-25 12:23:52,588 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-07-25 12:23:52,588 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-07-25 12:23:52,594 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-07-25 12:23:52,594 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-07-25 12:23:52,595 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-07-25 12:23:52,595 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-07-25 12:23:52,596 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-07-25 12:23:52,596 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-07-25 12:23:52,597 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-07-25 12:23:52,597 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-07-25 12:23:52,598 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-07-25 12:23:52,598 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10".
>2014-07-25 12:23:52,599 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-07-25 12:23:52,599 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-07-25 12:23:52,603 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-07-25 12:23:52,604 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt".
>2014-07-25 12:23:52,604 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-07-25 12:23:52,604 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-07-25 12:23:52,605 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-07-25 12:23:52,608 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-07-25 12:23:52,609 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-07-25 12:23:52,612 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-07-25 12:23:53,246 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend
>2014-07-25 12:23:53,744 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-07-25 12:23:53,842 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140722232058.git8e1babc.el6
>2014-07-25 12:23:53,843 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries
>2014-07-25 12:23:53,845 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries
>2014-07-25 12:23:53,846 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager
>2014-07-25 12:23:53,847 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-07-25 12:23:53,866 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager
>2014-07-25 12:23:53,867 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager
>2014-07-25 12:23:53,867 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-07-25 12:23:53,869 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager
>2014-07-25 12:23:53,871 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector
>2014-07-25 12:23:53,879 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree
>2014-07-25 12:23:53,887 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector
>2014-07-25 12:23:53,887 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer
>2014-07-25 12:23:53,895 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer
>2014-07-25 12:23:53,980 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers
>2014-07-25 12:23:54,005 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-07-25 12:23:54,017 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-07-25 12:23:54,019 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-07-25 12:23:54,019 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-07-25 12:23:54,020 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-07-25 12:23:54,020 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN
>2014-07-25 12:23:54,048 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager
>2014-07-25 12:23:54,049 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager
>2014-07-25 12:23:54,050 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager
>2014-07-25 12:23:54,052 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager
>2014-07-25 12:23:54,054 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector
>2014-07-25 12:23:54,056 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector
>2014-07-25 12:23:54,119 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list
>2014-07-25 12:23:54,119 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-07-25 12:23:54,120 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-07-25 12:23:54,121 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list
>2014-07-25 12:23:54,197 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager
>2014-07-25 12:23:54,206 INFO  [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager
>2014-07-25 12:23:54,241 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully.
>2014-07-25 12:23:54,244 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager
>2014-07-25 12:23:54,261 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-07-25 12:23:54,335 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Entered VdsManager constructor
>2014-07-25 12:23:54,392 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Initialize vdsBroker (10.34.62.208,54,321)
>2014-07-25 12:23:54,551 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-07-25 12:23:54,574 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager
>2014-07-25 12:23:54,579 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully.
>2014-07-25 12:23:54,580 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer
>2014-07-25 12:23:54,580 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer
>2014-07-25 12:23:54,581 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check
>2014-07-25 12:23:54,582 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check
>2014-07-25 12:23:54,594 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities
>2014-07-25 12:23:54,594 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities
>2014-07-25 12:23:54,599 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-07-25 12:23:54,604 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager
>2014-07-25 12:23:54,619 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255
>2014-07-25 12:23:54,632 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped
>2014-07-25 12:23:54,632 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager
>2014-07-25 12:23:54,633 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat
>2014-07-25 12:23:54,641 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized
>2014-07-25 12:23:58,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 62883ae1
>2014-07-25 12:23:58,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 62883ae1
>2014-07-25 12:23:58,765 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-07-25 12:23:58,942 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-25 12:29:09,183 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false.
>2014-07-25 12:29:09,196 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-07-25 12:29:09,206 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: UNASSIGNED not exist in string table
>2014-07-25 12:29:09,207 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-07-25 12:29:09,208 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-07-25 12:29:09,209 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-07-25 12:29:09,210 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-07-25 12:29:09,212 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-07-25 12:29:09,213 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-07-25 12:29:09,215 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-07-25 12:29:09,216 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-07-25 12:29:09,217 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-07-25 12:29:09,218 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_STOPPED not exist in string table
>2014-07-25 12:29:09,219 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_STARTED not exist in string table
>2014-07-25 12:29:09,220 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_ERROR not exist in string table
>2014-07-25 12:29:09,221 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-07-25 12:29:09,222 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-07-25 12:29:09,223 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-07-25 12:29:09,224 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-07-25 12:29:09,488 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-25 12:29:12,631 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-25 12:30:52,060 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35 value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-25 12:30:52,095 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 12:30:52,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2973de9b
>2014-07-25 12:30:52,139 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 2973de9b
>2014-07-25 12:30:52,182 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35 value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-25 12:30:52,432 INFO  [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-4) [35788b99] Running command: AddNFSStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 12:30:52,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5454a121
>2014-07-25 12:30:52,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5454a121
>2014-07-25 12:30:52,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@89784790, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35), log id: 6fc439b3
>2014-07-25 12:31:53,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, CreateStorageDomainVDSCommand, log id: 6fc439b3
>2014-07-25 12:31:53,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: bbd76bf
>2014-07-25 12:31:53,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@cbe9b7bb, log id: bbd76bf
>2014-07-25 12:31:53,755 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [35788b99] Correlation ID: 35788b99, Job ID: 9eef15b1-c4df-4808-a278-4bc7fbf0e982, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 was added by admin
>2014-07-25 12:31:53,926 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [b7be84c] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 12:31:53,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [b7be84c] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c915b93
>2014-07-25 12:31:54,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [b7be84c] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4c915b93
>2014-07-25 12:31:54,291 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-5) [64985a20] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:31:54,378 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [1d8b9dca] Lock Acquired to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>, sharedLocks= ]
>2014-07-25 12:31:54,392 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [1d8b9dca] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 12:31:54,524 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-5) [2afe048e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 12:31:54,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c37d8f7
>2014-07-25 12:31:54,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 7c37d8f7
>2014-07-25 12:31:54,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] START, CreateStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000002-0002-0002-0002-0000000001d9, storagePoolName=Default, masterDomainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, domainsIdList=[71f82fcd-4fa6-4f56-b228-f4fa4bd4f752], masterVersion=1), log id: 5c6286d1
>2014-07-25 12:33:18,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] FINISH, CreateStoragePoolVDSCommand, log id: 5c6286d1
>2014-07-25 12:33:18,138 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [2afe048e] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>, sharedLocks= ]
>2014-07-25 12:33:18,144 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-07-25 12:33:18,150 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:33:18,170 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-07-25 12:33:18,171 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/25/14 12:33 PM
>2014-07-25 12:33:18,180 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 12:33:18,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1ba54ccc
>2014-07-25 12:33:18,198 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 1ba54ccc
>2014-07-25 12:33:18,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-5) [714136e2] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 27f06ddd
>2014-07-25 12:33:18,242 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp--127.0.0.1-8702-5) [2169d51c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 12:33:18,267 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 2169d51c, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-07-25 12:33:18,278 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool Default
>2014-07-25 12:33:18,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 79b787a3
>2014-07-25 12:33:26,468 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false.
>2014-07-25 12:33:26,473 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-25 12:33:26,542 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-07-25 12:34:18,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, ConnectStoragePoolVDSCommand, log id: 79b787a3
>2014-07-25 12:34:18,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-25 12:34:18,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-25 12:34:18,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b28ef2d
>2014-07-25 12:34:18,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling started: taskId = 7ac72fd0-25e2-484b-8110-f38874cec8db
>2014-07-25 12:34:19,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling ended: taskId = 7ac72fd0-25e2-484b-8110-f38874cec8db task status = finished
>2014-07-25 12:34:19,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling ended, spm status: SPM
>2014-07-25 12:34:19,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ac72fd0-25e2-484b-8110-f38874cec8db), log id: 766a4f98
>2014-07-25 12:34:19,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, HSMClearTaskVDSCommand, log id: 766a4f98
>2014-07-25 12:34:19,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23d03910, log id: 1b28ef2d
>2014-07-25 12:34:19,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-25 12:34:19,988 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-25 12:34:20,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, ActivateStorageDomainVDSCommand, log id: 27f06ddd
>2014-07-25 12:34:20,086 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [2169d51c] ActivateStorage Domain. After Connect all hosts to pool. Time:7/25/14 12:34 PM
>2014-07-25 12:34:20,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3dc9c0e6
>2014-07-25 12:34:51,876 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Lock Acquired to object EngineLock [exclusiveLocks= key: blank-disk value: VM_NAME
>, sharedLocks= ]
>2014-07-25 12:34:51,932 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Running command: AddVmFromScratchCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-25 12:34:52,150 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4ad76509] Correlation ID: 4ad76509, Job ID: fbbbb9f6-5de6-4768-81a6-e8788a231053, Call Stack: null, Custom Event ID: -1, Message: VM blank-disk was created by admin.
>2014-07-25 12:34:52,162 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Lock freed to object EngineLock [exclusiveLocks= key: blank-disk value: VM_NAME
>, sharedLocks= ]
>2014-07-25 12:34:52,809 INFO  [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-6) [27b402b3] Running command: AddVmInterfaceCommand internal: false. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM,  ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile
>2014-07-25 12:34:52,997 INFO  [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-6) [3fedec69] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM
>2014-07-25 12:34:53,036 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3fedec69] Correlation ID: 3fedec69, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM blank-disk. (User: admin)
>2014-07-25 12:34:53,056 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3fedec69] Correlation ID: 27b402b3, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM blank-disk. (User: admin)
>2014-07-25 12:34:53,289 INFO  [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-1) [330fdfc7] Running command: ReorderVmNicsCommand internal: false. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM
>2014-07-25 12:34:55,926 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [1c32d79d] Lock Acquired to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM_DISK_BOOT
>, sharedLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM
>]
>2014-07-25 12:34:55,961 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [1c32d79d] Running command: AddDiskCommand internal: false. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:34:56,138 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:34:56,178 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] Lock freed to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM_DISK_BOOT
>, sharedLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM
>]
>2014-07-25 12:34:56,182 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageSizeInBytes = 1073741824, volumeFormat = RAW, newImageId = 1530156b-4bdd-4521-8646-af3e136ea23d, newImageDescription = ), log id: 6fdb0682
>2014-07-25 12:34:56,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-25 12:34:56,616 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] FINISH, CreateImageVDSCommand, return: 1530156b-4bdd-4521-8646-af3e136ea23d, log id: 6fdb0682
>2014-07-25 12:34:56,890 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [1637ac6c] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4
>2014-07-25 12:34:56,891 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [1637ac6c] CommandMultiAsyncTasks::AttachTask: Attaching task 030be765-71d2-41c6-ae9d-1c1d80564bc8 to command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4.
>2014-07-25 12:34:56,919 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [1637ac6c] Adding task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:34:56,977 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1637ac6c] Correlation ID: 1c32d79d, Job ID: 6627e838-dd49-43d2-b7cd-bb7961827f15, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of blank-disk_Disk1 was initiated on VM blank-disk by admin.
>2014-07-25 12:34:56,978 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) [1637ac6c] BaseAsyncTask::startPollingTask: Starting to poll task 030be765-71d2-41c6-ae9d-1c1d80564bc8.
>2014-07-25 12:35:04,231 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-93) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-07-25 12:35:04,241 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) SPMAsyncTask::PollTask: Polling task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:35:04,259 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) BaseAsyncTask::OnTaskEndSuccess: Task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:35:04,263 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::EndActionIfNecessary: All tasks of command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4 has ended -> executing endAction
>2014-07-25 12:35:04,265 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 850cd3e1-f5e5-4ce5-9516-7f03572f46a4): calling endAction .
>2014-07-25 12:35:04,268 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0
>2014-07-25 12:35:04,307 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-5) [1c32d79d] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand
>2014-07-25 12:35:04,329 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-25 12:35:04,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageId = 1530156b-4bdd-4521-8646-af3e136ea23d), log id: 1cb6546d
>2014-07-25 12:35:04,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@5347c20d, log id: 1cb6546d
>2014-07-25 12:35:04,440 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) Correlation ID: 1c32d79d, Call Stack: null, Custom Event ID: -1, Message: The disk blank-disk_Disk1 was successfully added to VM blank-disk.
>2014-07-25 12:35:04,442 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result.
>2014-07-25 12:35:04,443 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks.
>2014-07-25 12:35:04,447 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-5) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 030be765-71d2-41c6-ae9d-1c1d80564bc8
>2014-07-25 12:35:04,450 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 030be765-71d2-41c6-ae9d-1c1d80564bc8), log id: 7c2a50d4
>2014-07-25 12:35:04,458 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=030be765-71d2-41c6-ae9d-1c1d80564bc8), log id: 3e20cdfb
>2014-07-25 12:35:04,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, HSMClearTaskVDSCommand, log id: 3e20cdfb
>2014-07-25 12:35:04,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, SPMClearTaskVDSCommand, log id: 7c2a50d4
>2014-07-25 12:35:04,579 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-5) BaseAsyncTask::RemoveTaskFromDB: Removed task 030be765-71d2-41c6-ae9d-1c1d80564bc8 from DataBase
>2014-07-25 12:35:04,580 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 850cd3e1-f5e5-4ce5-9516-7f03572f46a4
>2014-07-25 12:35:06,987 INFO  [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp--127.0.0.1-8702-7) Running command: AddVmTemplateCommand internal: false. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 12:35:06,995 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = ImageLocked, exit status = Normal), log id: 4c36b646
>2014-07-25 12:35:07,009 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVmStatusVDSCommand, log id: 4c36b646
>2014-07-25 12:35:07,042 INFO  [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] Running command: CreateImageTemplateCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:35:07,061 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageId = 1530156b-4bdd-4521-8646-af3e136ea23d, dstImageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, dstImageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00, imageDescription = Active VM, dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 69d3dbe7
>2014-07-25 12:35:07,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID
>2014-07-25 12:35:07,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] -- copyImage parameters:
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                vmGUID=a69ec17d-4439-4965-84d0-5a99ccfb596b
>                srcImageGUID=ca914369-e619-456d-8e1e-0459ff4a54cd
>                srcVolUUID=1530156b-4bdd-4521-8646-af3e136ea23d
>                dstImageGUID=4ace1a06-1fa4-4d9f-8442-fc3e2a3def19
>                dstVolUUID=ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00
>                descr=Active VM
>                dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-25 12:35:07,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 69d3dbe7
>2014-07-25 12:35:07,455 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [a8d66fd] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command a0a42c45-c1b5-4aef-b4be-7addcb109965
>2014-07-25 12:35:07,456 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [a8d66fd] CommandMultiAsyncTasks::AttachTask: Attaching task 5e2729b7-f255-4396-ad79-5e7008d4fede to command a0a42c45-c1b5-4aef-b4be-7addcb109965.
>2014-07-25 12:35:07,469 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [a8d66fd] Adding task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:35:07,520 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 2e241ac8, Job ID: 44319d01-a69a-41ad-bbd1-092f24b63721, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blank-disk from VM blank-disk was initiated by admin.
>2014-07-25 12:35:07,522 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) BaseAsyncTask::startPollingTask: Starting to poll task 5e2729b7-f255-4396-ad79-5e7008d4fede.
>2014-07-25 12:35:14,267 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now
>2014-07-25 12:35:14,275 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) SPMAsyncTask::PollTask: Polling task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 12:35:14,276 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) Finished polling Tasks, will poll again in 10 seconds.
>2014-07-25 12:35:20,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, ConnectStoragePoolVDSCommand, log id: 3dc9c0e6
>2014-07-25 12:35:20,360 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [2169d51c] ActivateStorage Domain. After change storage pool status in vds. Time:7/25/14 12:35 PM
>2014-07-25 12:35:20,373 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 714136e2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 (Data Center Default) was activated by admin
>2014-07-25 12:35:20,385 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 1d8b9dca, Job ID: c2b818e8-7fd1-490a-aa85-4bbfcfb35b60, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center Default by admin
>2014-07-25 12:35:20,403 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 64985a20, Job ID: c2b818e8-7fd1-490a-aa85-4bbfcfb35b60, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 was attached to Data Center Default by admin
>2014-07-25 12:35:24,285 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-33) SPMAsyncTask::PollTask: Polling task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:35:24,292 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-33) BaseAsyncTask::OnTaskEndSuccess: Task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:35:24,294 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-33) CommandAsyncTask::EndActionIfNecessary: All tasks of command a0a42c45-c1b5-4aef-b4be-7addcb109965 has ended -> executing endAction
>2014-07-25 12:35:24,295 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-33) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: a0a42c45-c1b5-4aef-b4be-7addcb109965): calling endAction .
>2014-07-25 12:35:24,297 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmTemplate, executionIndex: 0
>2014-07-25 12:35:24,317 INFO  [org.ovirt.engine.core.bll.AddVmTemplateCommand] (org.ovirt.thread.pool-8-thread-19) [2e241ac8] Ending command successfully: org.ovirt.engine.core.bll.AddVmTemplateCommand
>2014-07-25 12:35:24,321 INFO  [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] Ending command successfully: org.ovirt.engine.core.bll.CreateImageTemplateCommand
>2014-07-25 12:35:24,324 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, imageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00), log id: 5d3c0c24
>2014-07-25 12:35:24,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e0cc3b16, log id: 5d3c0c24
>2014-07-25 12:35:24,368 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = Down, exit status = Normal), log id: 780f346
>2014-07-25 12:35:24,376 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SetVmStatusVDSCommand, log id: 780f346
>2014-07-25 12:35:24,402 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) Correlation ID: 2e241ac8, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blank-disk from VM blank-disk has been completed.
>2014-07-25 12:35:24,403 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate completed, handling the result.
>2014-07-25 12:35:24,404 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate succeeded, clearing tasks.
>2014-07-25 12:35:24,409 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-19) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 5e2729b7-f255-4396-ad79-5e7008d4fede
>2014-07-25 12:35:24,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 5e2729b7-f255-4396-ad79-5e7008d4fede), log id: 20f87b6b
>2014-07-25 12:35:24,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5e2729b7-f255-4396-ad79-5e7008d4fede), log id: 35aaabc5
>2014-07-25 12:35:24,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, HSMClearTaskVDSCommand, log id: 35aaabc5
>2014-07-25 12:35:24,474 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SPMClearTaskVDSCommand, log id: 20f87b6b
>2014-07-25 12:35:24,484 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-19) BaseAsyncTask::RemoveTaskFromDB: Removed task 5e2729b7-f255-4396-ad79-5e7008d4fede from DataBase
>2014-07-25 12:35:24,485 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity a0a42c45-c1b5-4aef-b4be-7addcb109965
>2014-07-25 12:35:52,521 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-6) [52c9f8ef] Lock Acquired to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM
>, sharedLocks= ]
>2014-07-25 12:35:52,562 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] Running command: RemoveVmCommand internal: false. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM
>2014-07-25 12:35:52,564 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = ImageLocked, exit status = Normal), log id: 75b3a6f6
>2014-07-25 12:35:52,571 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] FINISH, SetVmStatusVDSCommand, log id: 75b3a6f6
>2014-07-25 12:35:52,575 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] Lock freed to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM
>, sharedLocks= ]
>2014-07-25 12:35:52,631 INFO  [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-6) [20bcf485] Running command: RemoveAllVmImagesCommand internal: true. Entities affected :  ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM
>2014-07-25 12:35:52,647 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Running command: RemoveImageCommand internal: true. Entities affected :  ID: 00000000-0000-0000-0000-000000000000 Type: Storage
>2014-07-25 12:35:52,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, postZeros = false, forceDelete = false), log id: a158fda
>2014-07-25 12:35:53,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] FINISH, DeleteImageGroupVDSCommand, log id: a158fda
>2014-07-25 12:35:53,523 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 9efb0336-24c0-4844-895e-ef33b3f65b8c
>2014-07-25 12:35:53,525 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] CommandMultiAsyncTasks::AttachTask: Attaching task 881c70d7-3bff-4f74-9708-93c6af989306 to command 9efb0336-24c0-4844-895e-ef33b3f65b8c.
>2014-07-25 12:35:53,555 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Adding task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:35:53,826 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Correlation ID: 52c9f8ef, Job ID: 3e0c1f99-0524-447e-ad14-18c9e753376f, Call Stack: null, Custom Event ID: -1, Message: VM blank-disk was successfully removed.
>2014-07-25 12:35:53,827 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] BaseAsyncTask::startPollingTask: Starting to poll task 881c70d7-3bff-4f74-9708-93c6af989306.
>2014-07-25 12:35:54,298 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now
>2014-07-25 12:35:54,305 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 12:35:54,307 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Finished polling Tasks, will poll again in 10 seconds.
>2014-07-25 12:36:04,314 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-24) SPMAsyncTask::PollTask: Polling task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:36:04,325 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-24) BaseAsyncTask::OnTaskEndSuccess: Task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:36:04,327 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-24) CommandAsyncTask::EndActionIfNecessary: All tasks of command 9efb0336-24c0-4844-895e-ef33b3f65b8c has ended -> executing endAction
>2014-07-25 12:36:04,328 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-24) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 9efb0336-24c0-4844-895e-ef33b3f65b8c): calling endAction .
>2014-07-25 12:36:04,330 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0
>2014-07-25 12:36:04,343 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-16) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand
>2014-07-25 12:36:04,345 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result.
>2014-07-25 12:36:04,346 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks.
>2014-07-25 12:36:04,351 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-16) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 881c70d7-3bff-4f74-9708-93c6af989306
>2014-07-25 12:36:04,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 881c70d7-3bff-4f74-9708-93c6af989306), log id: 3994544b
>2014-07-25 12:36:04,357 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=881c70d7-3bff-4f74-9708-93c6af989306), log id: 7e0141a4
>2014-07-25 12:36:07,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMClearTaskVDSCommand, log id: 7e0141a4
>2014-07-25 12:36:07,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMClearTaskVDSCommand, log id: 3994544b
>2014-07-25 12:36:07,685 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-16) BaseAsyncTask::RemoveTaskFromDB: Removed task 881c70d7-3bff-4f74-9708-93c6af989306 from DataBase
>2014-07-25 12:36:07,686 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 9efb0336-24c0-4844-895e-ef33b3f65b8c
>2014-07-25 12:36:23,644 INFO  [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] Lock Acquired to object EngineLock [exclusiveLocks= key: runit value: VM_NAME
>, sharedLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE
>key: 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 value: DISK
>]
>2014-07-25 12:36:23,709 INFO  [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] Running command: AddVmFromTemplateCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups,  ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:36:23,745 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = ImageLocked, exit status = Normal), log id: 56d513e9
>2014-07-25 12:36:23,750 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] FINISH, SetVmStatusVDSCommand, log id: 56d513e9
>2014-07-25 12:36:23,885 INFO  [org.ovirt.engine.core.bll.CreateCloneOfTemplateCommand] (ajp--127.0.0.1-8702-3) [76f353cc] Running command: CreateCloneOfTemplateCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:36:23,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, imageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00, dstImageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, vmId = d031b0a2-efae-4300-b703-af169651be32, dstImageId = 827408c6-d52c-4c49-92f6-461ce222e485, imageDescription = , dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 75999916
>2014-07-25 12:36:23,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID
>2014-07-25 12:36:23,976 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] -- copyImage parameters:
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                vmGUID=d031b0a2-efae-4300-b703-af169651be32
>                srcImageGUID=4ace1a06-1fa4-4d9f-8442-fc3e2a3def19
>                srcVolUUID=ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00
>                dstImageGUID=1f7ff79b-b034-47ad-8864-181be4e1de5a
>                dstVolUUID=827408c6-d52c-4c49-92f6-461ce222e485
>                descr=
>                dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-25 12:36:24,227 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-66) Setting new tasks map. The map contains now 2 tasks
>2014-07-25 12:36:25,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 75999916
>2014-07-25 12:36:25,760 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [76f353cc] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 37d356ed-5485-42b9-b009-43ec0801ae2b
>2014-07-25 12:36:25,761 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [76f353cc] CommandMultiAsyncTasks::AttachTask: Attaching task c3598f13-53e7-408c-8e69-ec66e386df0f to command 37d356ed-5485-42b9-b009-43ec0801ae2b.
>2014-07-25 12:36:25,802 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [76f353cc] Adding task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:36:25,904 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [76f353cc] Correlation ID: 1ab40ee9, Job ID: 50b98694-8560-4821-b756-9c4d4e038a15, Call Stack: null, Custom Event ID: -1, Message: VM runit creation was initiated by admin.
>2014-07-25 12:36:25,906 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [76f353cc] BaseAsyncTask::startPollingTask: Starting to poll task c3598f13-53e7-408c-8e69-ec66e386df0f.
>2014-07-25 12:36:26,375 INFO  [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-6) [3de12490] Running command: ReorderVmNicsCommand internal: false. Entities affected :  ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM
>2014-07-25 12:36:34,336 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now
>2014-07-25 12:36:34,344 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 12:36:34,345 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Finished polling Tasks, will poll again in 10 seconds.
>2014-07-25 12:36:44,353 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) SPMAsyncTask::PollTask: Polling task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:36:44,362 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) BaseAsyncTask::OnTaskEndSuccess: Task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:36:44,371 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) CommandAsyncTask::EndActionIfNecessary: All tasks of command 37d356ed-5485-42b9-b009-43ec0801ae2b has ended -> executing endAction
>2014-07-25 12:36:44,373 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 37d356ed-5485-42b9-b009-43ec0801ae2b): calling endAction .
>2014-07-25 12:36:44,374 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmFromTemplate, executionIndex: 0
>2014-07-25 12:36:44,420 INFO  [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) [1ab40ee9] Ending command successfully: org.ovirt.engine.core.bll.AddVmFromTemplateCommand
>2014-07-25 12:36:44,434 INFO  [org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] Ending command successfully: org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand
>2014-07-25 12:36:44,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, imageId = 827408c6-d52c-4c49-92f6-461ce222e485), log id: 27b3c6c3
>2014-07-25 12:36:44,476 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@2a00c8d9, log id: 27b3c6c3
>2014-07-25 12:36:44,483 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = Down, exit status = Normal), log id: 12c5b010
>2014-07-25 12:36:44,488 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, SetVmStatusVDSCommand, log id: 12c5b010
>2014-07-25 12:36:44,491 INFO  [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) Lock freed to object EngineLock [exclusiveLocks= key: runit value: VM_NAME
>, sharedLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE
>key: 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 value: DISK
>]
>2014-07-25 12:36:44,502 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) Correlation ID: 1ab40ee9, Call Stack: null, Custom Event ID: -1, Message: VM runit creation has been completed.
>2014-07-25 12:36:44,503 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromTemplate completed, handling the result.
>2014-07-25 12:36:44,505 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromTemplate succeeded, clearing tasks.
>2014-07-25 12:36:44,510 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-14) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c3598f13-53e7-408c-8e69-ec66e386df0f
>2014-07-25 12:36:44,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = c3598f13-53e7-408c-8e69-ec66e386df0f), log id: 7d6b229e
>2014-07-25 12:36:44,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c3598f13-53e7-408c-8e69-ec66e386df0f), log id: 4adf366
>2014-07-25 12:36:44,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, HSMClearTaskVDSCommand, log id: 4adf366
>2014-07-25 12:36:44,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, SPMClearTaskVDSCommand, log id: 7d6b229e
>2014-07-25 12:36:44,586 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-14) BaseAsyncTask::RemoveTaskFromDB: Removed task c3598f13-53e7-408c-8e69-ec66e386df0f from DataBase
>2014-07-25 12:36:44,587 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 37d356ed-5485-42b9-b009-43ec0801ae2b
>2014-07-25 12:36:48,727 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Lock Acquired to object EngineLock [exclusiveLocks= key: new-one value: VM_NAME
>, sharedLocks= ]
>2014-07-25 12:36:48,764 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Running command: AddVmFromScratchCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-25 12:36:48,833 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [35bad99] Correlation ID: 35bad99, Job ID: dc49a405-5464-4301-91b7-828c687b2d7b, Call Stack: null, Custom Event ID: -1, Message: VM new-one was created by admin.
>2014-07-25 12:36:48,839 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Lock freed to object EngineLock [exclusiveLocks= key: new-one value: VM_NAME
>, sharedLocks= ]
>2014-07-25 12:36:49,418 INFO  [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-4) [54bc166a] Running command: AddVmInterfaceCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM,  ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile
>2014-07-25 12:36:49,448 INFO  [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-4) [34d4de61] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:36:49,468 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [34d4de61] Correlation ID: 34d4de61, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM new-one. (User: admin)
>2014-07-25 12:36:49,478 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [34d4de61] Correlation ID: 54bc166a, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM new-one. (User: admin)
>2014-07-25 12:36:49,570 INFO  [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-6) [5c934e7c] Running command: ReorderVmNicsCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:36:52,489 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [a186681] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM_DISK_BOOT
>, sharedLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>]
>2014-07-25 12:36:52,516 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [a186681] Running command: AddDiskCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:36:52,545 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [1338c46b] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:36:52,570 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [1338c46b] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM_DISK_BOOT
>, sharedLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>]
>2014-07-25 12:36:52,574 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, imageSizeInBytes = 1073741824, volumeFormat = RAW, newImageId = 860b11ae-d1a7-4c83-ad10-4989fcca0638, newImageDescription = ), log id: 1487e536
>2014-07-25 12:36:52,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-25 12:36:54,228 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-33) Setting new tasks map. The map contains now 2 tasks
>2014-07-25 12:36:59,133 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] FINISH, CreateImageVDSCommand, return: 860b11ae-d1a7-4c83-ad10-4989fcca0638, log id: 1487e536
>2014-07-25 12:36:59,142 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-2) [1338c46b] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7
>2014-07-25 12:36:59,145 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-2) [1338c46b] CommandMultiAsyncTasks::AttachTask: Attaching task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 to command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7.
>2014-07-25 12:36:59,177 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-2) [1338c46b] Adding task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:36:59,208 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1338c46b] Correlation ID: a186681, Job ID: c446e85e-37e1-41fd-8915-a9c0eff06428, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of new-one_Disk1 was initiated on VM new-one by admin.
>2014-07-25 12:36:59,209 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-2) [1338c46b] BaseAsyncTask::startPollingTask: Starting to poll task 7ac7f77c-031c-4313-b1ee-0acaf9345c00.
>2014-07-25 12:37:04,376 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now
>2014-07-25 12:37:04,383 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) SPMAsyncTask::PollTask: Polling task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 12:37:04,386 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) Finished polling Tasks, will poll again in 10 seconds.
>2014-07-25 12:37:05,333 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:05,385 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [2abfccd] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 519a9fe8
>2014-07-25 12:37:05,386 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [2abfccd] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 519a9fe8
>2014-07-25 12:37:05,390 WARN  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISKS_LOCKED,$diskAliases new-one_Disk1
>2014-07-25 12:37:05,391 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:13,163 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:13,197 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-7) [85e122a] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 2f8fc05b
>2014-07-25 12:37:13,199 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-7) [85e122a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2f8fc05b
>2014-07-25 12:37:13,202 WARN  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISKS_LOCKED,$diskAliases new-one_Disk1
>2014-07-25 12:37:13,203 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:14,393 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-78) SPMAsyncTask::PollTask: Polling task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:37:14,401 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-78) BaseAsyncTask::OnTaskEndSuccess: Task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:37:14,403 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-78) CommandAsyncTask::EndActionIfNecessary: All tasks of command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7 has ended -> executing endAction
>2014-07-25 12:37:14,405 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-78) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7): calling endAction .
>2014-07-25 12:37:14,406 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0
>2014-07-25 12:37:14,417 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-7) [a186681] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand
>2014-07-25 12:37:14,435 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-25 12:37:14,460 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, imageId = 860b11ae-d1a7-4c83-ad10-4989fcca0638), log id: 21e9968d
>2014-07-25 12:37:14,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e991043e, log id: 21e9968d
>2014-07-25 12:37:14,513 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-7) Correlation ID: a186681, Call Stack: null, Custom Event ID: -1, Message: The disk new-one_Disk1 was successfully added to VM new-one.
>2014-07-25 12:37:14,514 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result.
>2014-07-25 12:37:14,516 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks.
>2014-07-25 12:37:14,520 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-7) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7ac7f77c-031c-4313-b1ee-0acaf9345c00
>2014-07-25 12:37:14,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 7ac7f77c-031c-4313-b1ee-0acaf9345c00), log id: 34d9da9f
>2014-07-25 12:37:14,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ac7f77c-031c-4313-b1ee-0acaf9345c00), log id: 4aa017d4
>2014-07-25 12:37:14,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, HSMClearTaskVDSCommand, log id: 4aa017d4
>2014-07-25 12:37:14,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, SPMClearTaskVDSCommand, log id: 34d9da9f
>2014-07-25 12:37:14,594 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-7) BaseAsyncTask::RemoveTaskFromDB: Removed task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 from DataBase
>2014-07-25 12:37:14,596 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7
>2014-07-25 12:37:24,004 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [48ba118e] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:24,107 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [48ba118e] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 3d6eef82
>2014-07-25 12:37:24,108 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [48ba118e] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3d6eef82
>2014-07-25 12:37:24,232 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Setting new tasks map. The map contains now 2 tasks
>2014-07-25 12:37:24,318 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Running command: RunVmCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:37:24,510 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@e46273a5), log id: 7e7740f
>2014-07-25 12:37:24,516 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7e7740f
>2014-07-25 12:37:24,535 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, vm=VM [new-one]), log id: 1498ae3e
>2014-07-25 12:37:24,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, vm=VM [new-one]), log id: 598c9d44
>2014-07-25 12:37:24,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs
>CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog
>ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0
>Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiMTI0M2Y4ZjgtN2U3MS00NTFj
>LWI1M2QtYWViZmJiOWQyOThlIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz
>ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ==
>, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg
>YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp
>cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0
>Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg
>IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK
>, openstack/content/0000=YXV0bwo=
>}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=fc9306e3-ca33-4e96-9f08-c822bb1ee57c}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=81860101-8c00-4aa1-9d32-bdc52a009f2c}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8e29dccb-8a92-43ab-94f4-d0ec7540dedf}, {shared=false, volumeID=860b11ae-d1a7-4c83-ad10-4989fcca0638, index=0, iface=virtio, imageID=43a334b9-1e3f-472c-b32e-e1ed0c074c14, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=43a334b9-1e3f-472c-b32e-e1ed0c074c14}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=6b19a1fd-543e-41f4-93cf-9d406414ea8e}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=1f6ab695-7327-4787-aa54-f472d3e37b60}],vmName=new-one,cpuType=Westmere,fileTransferEnable=true
>2014-07-25 12:37:24,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, CreateVmFromCloudInitVDSCommand, log id: 598c9d44
>2014-07-25 12:37:24,809 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1498ae3e
>2014-07-25 12:37:24,810 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:37:24,828 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Correlation ID: 48ba118e, Job ID: 3d6d18b7-2f82-41b0-bf6c-e4a5d6a91e70, Call Stack: null, Custom Event ID: -1, Message: VM new-one was started by admin (Host: rack).
>2014-07-25 12:37:28,006 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) VM new-one 6f56ce08-46a9-4bfe-b207-50fd71e003cc moved from WaitForLaunch --> PoweringUp
>2014-07-25 12:37:28,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[6f56ce08-46a9-4bfe-b207-50fd71e003cc]), log id: 720b0f07
>2014-07-25 12:37:28,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=18551, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@3407cf0f, vmName=new-one, fileTransferEnable=true, cpuType=Westmere}], log id: 720b0f07
>2014-07-25 12:37:28,042 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Received a memballoon Device without an address when processing VM 6f56ce08-46a9-4bfe-b207-50fd71e003cc devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-25 12:37:28,044 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Received a spice Device without an address when processing VM 6f56ce08-46a9-4bfe-b207-50fd71e003cc devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-25 12:37:36,023 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] Lock Acquired to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM
>, sharedLocks= ]
>2014-07-25 12:37:36,122 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] START, IsVmDuringInitiatingVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db), log id: 2b3a75aa
>2014-07-25 12:37:36,123 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2b3a75aa
>2014-07-25 12:37:36,220 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Running command: RunVmCommand internal: false. Entities affected :  ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM
>2014-07-25 12:37:36,273 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@f168cd40), log id: 713ee3ec
>2014-07-25 12:37:36,281 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, UpdateVmDynamicDataVDSCommand, log id: 713ee3ec
>2014-07-25 12:37:36,293 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, vm=VM [runit]), log id: 5a3b3c0d
>2014-07-25 12:37:36,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, vm=VM [runit]), log id: 590a9463
>2014-07-25 12:37:36,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs
>CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog
>ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0
>Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiNDAyZjY1ZTMtZTg4My00ZTE0
>LTkxNjItOTVjZjAzODg5MzY0IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz
>ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ==
>, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg
>YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp
>cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0
>Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg
>IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK
>, openstack/content/0000=YXV0bwo=
>}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=7926ee2a-7604-43c1-9ec3-86ec9408b946}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=2580cdcf-a605-42d7-8773-966ee1226d36}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=326811c5-8ddf-4c9c-be7d-7c994b7a8c19}, {shared=false, index=0, volumeID=827408c6-d52c-4c49-92f6-461ce222e485, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=1f7ff79b-b034-47ad-8864-181be4e1de5a, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=1f7ff79b-b034-47ad-8864-181be4e1de5a}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=7341cbff-1885-4900-b72f-29066e4a550d}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=08ab4b0f-3cf3-4f84-a470-cee120c62a26}],vmName=runit,cpuType=Westmere,fileTransferEnable=true
>2014-07-25 12:37:36,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, CreateVmFromCloudInitVDSCommand, log id: 590a9463
>2014-07-25 12:37:36,438 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5a3b3c0d
>2014-07-25 12:37:36,439 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Lock freed to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM
>, sharedLocks= ]
>2014-07-25 12:37:36,452 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Correlation ID: 5cdd9d25, Job ID: 98a33085-e41a-4c6a-830b-138aaa35f2e6, Call Stack: null, Custom Event ID: -1, Message: VM runit was started by admin (Host: rack).
>2014-07-25 12:37:37,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[f7613d6f-8091-4d21-90e6-c13219a1b3db]), log id: 5696275e
>2014-07-25 12:37:37,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=18709, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@5c3e0b52, vmName=runit, fileTransferEnable=true, cpuType=Westmere}], log id: 5696275e
>2014-07-25 12:37:37,294 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a memballoon Device without an address when processing VM f7613d6f-8091-4d21-90e6-c13219a1b3db devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-25 12:37:37,296 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a spice Device without an address when processing VM f7613d6f-8091-4d21-90e6-c13219a1b3db devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5903, type=graphics}
>2014-07-25 12:37:40,388 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) VM runit f7613d6f-8091-4d21-90e6-c13219a1b3db moved from WaitForLaunch --> PoweringUp
>2014-07-25 12:37:54,233 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-67) Setting new tasks map. The map contains now 1 tasks
>2014-07-25 12:37:58,127 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-21) [5eb231f3] Start running CanDoAction for command number 1/2 (Command type: StopVm)
>2014-07-25 12:37:58,127 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-19) [4ed7917] Start running CanDoAction for command number 2/2 (Command type: StopVm)
>2014-07-25 12:37:58,154 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-19) [4ed7917] Finish handling CanDoAction for command number 2/2 (Command type: StopVm)
>2014-07-25 12:37:58,164 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-21) [5eb231f3] Finish handling CanDoAction for command number 1/2 (Command type: StopVm)
>2014-07-25 12:37:58,190 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] Running command: StopVmCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:37:58,200 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, force=false, secondsToWait=0, gracefully=false, reason=), log id: 328ee15c
>2014-07-25 12:37:58,224 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, force=false, secondsToWait=0, gracefully=false, reason=), log id: 2e156077
>2014-07-25 12:37:58,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] FINISH, DestroyVDSCommand, log id: 2e156077
>2014-07-25 12:37:58,652 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] FINISH, DestroyVmVDSCommand, return: Down, log id: 328ee15c
>2014-07-25 12:37:58,659 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-22) [259e5d4c] Running command: ProcessDownVmCommand internal: true.
>2014-07-25 12:37:58,667 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] Correlation ID: 5eb231f3, Job ID: 5f88f5c8-0263-4e61-bfb2-97fb00d16aa4, Call Stack: null, Custom Event ID: -1, Message: VM new-one powered off by admin (Host: rack) (Reason: Not Specified).
>2014-07-25 12:37:58,694 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] Running command: StopVmCommand internal: false. Entities affected :  ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM
>2014-07-25 12:37:58,699 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, force=false, secondsToWait=0, gracefully=false, reason=), log id: 763db38b
>2014-07-25 12:37:58,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, force=false, secondsToWait=0, gracefully=false, reason=), log id: 500c5c1a
>2014-07-25 12:37:59,123 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] FINISH, DestroyVDSCommand, log id: 500c5c1a
>2014-07-25 12:37:59,132 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] FINISH, DestroyVmVDSCommand, return: Down, log id: 763db38b
>2014-07-25 12:37:59,143 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-24) [d6d8943] Running command: ProcessDownVmCommand internal: true.
>2014-07-25 12:37:59,144 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [4ed7917] Correlation ID: 4ed7917, Job ID: 6fca0289-fce3-4613-ac1a-b6feaf49ad8f, Call Stack: null, Custom Event ID: -1, Message: VM runit powered off by admin (Host: rack) (Reason: Not Specified).
>2014-07-25 12:38:24,235 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-35) Setting new tasks map. The map contains now 0 tasks
>2014-07-25 12:38:24,236 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-35) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-25 12:39:01,909 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-25 12:39:01,910 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-47) [25552b90] Start running CanDoAction for command number 1/2 (Command type: RemoveVm)
>2014-07-25 12:39:01,911 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-48) [79983e87] Start running CanDoAction for command number 2/2 (Command type: RemoveVm)
>2014-07-25 12:39:01,916 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-48) [79983e87] Lock Acquired to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM
>, sharedLocks= ]
>2014-07-25 12:39:01,917 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-47) [25552b90] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:39:01,951 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-47) [25552b90] Finish handling CanDoAction for command number 1/2 (Command type: RemoveVm)
>2014-07-25 12:39:01,974 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-48) [79983e87] Finish handling CanDoAction for command number 2/2 (Command type: RemoveVm)
>2014-07-25 12:39:01,998 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] Running command: RemoveVmCommand internal: false. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:39:01,999 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] START, SetVmStatusVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc, status = ImageLocked, exit status = Normal), log id: 3792fa71
>2014-07-25 12:39:02,006 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] FINISH, SetVmStatusVDSCommand, log id: 3792fa71
>2014-07-25 12:39:02,010 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM
>, sharedLocks= ]
>2014-07-25 12:39:02,059 INFO  [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-49) [60eabe5a] Running command: RemoveAllVmImagesCommand internal: true. Entities affected :  ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM
>2014-07-25 12:39:02,072 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Running command: RemoveImageCommand internal: true. Entities affected :  ID: 00000000-0000-0000-0000-000000000000 Type: Storage
>2014-07-25 12:39:02,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, postZeros = false, forceDelete = false), log id: 149bdd1d
>2014-07-25 12:39:10,693 INFO  [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp--127.0.0.1-8702-2) [7eafb87d] Lock Acquired to object EngineLock [exclusiveLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE
>, sharedLocks= ]
>2014-07-25 12:39:10,785 INFO  [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-11) [7eafb87d] Running command: RemoveVmTemplateCommand internal: false. Entities affected :  ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate
>2014-07-25 12:39:10,806 INFO  [org.ovirt.engine.core.bll.RemoveAllVmTemplateImageTemplatesCommand] (org.ovirt.thread.pool-8-thread-11) [7eafb87d] Running command: RemoveAllVmTemplateImageTemplatesCommand internal: true. Entities affected :  ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate
>2014-07-25 12:39:10,827 INFO  [org.ovirt.engine.core.bll.RemoveTemplateSnapshotCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Running command: RemoveTemplateSnapshotCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 12:39:10,859 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, postZeros = false, forceDelete = false), log id: 68836761
>2014-07-25 12:39:12,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] FINISH, DeleteImageGroupVDSCommand, log id: 149bdd1d
>2014-07-25 12:39:12,440 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) [2de72aba] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a
>2014-07-25 12:39:12,441 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-49) [2de72aba] CommandMultiAsyncTasks::AttachTask: Attaching task db8a1b94-2090-4c17-b6ba-f686f03e2b01 to command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a.
>2014-07-25 12:39:12,453 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Adding task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:39:12,528 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Correlation ID: 25552b90, Job ID: 785d6e4c-b858-4f0f-b3d0-f38642c23434, Call Stack: null, Custom Event ID: -1, Message: VM new-one was successfully removed.
>2014-07-25 12:39:12,529 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) [2de72aba] BaseAsyncTask::startPollingTask: Starting to poll task db8a1b94-2090-4c17-b6ba-f686f03e2b01.
>2014-07-25 12:39:12,543 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] Running command: RemoveVmCommand internal: false. Entities affected :  ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM
>2014-07-25 12:39:12,544 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = ImageLocked, exit status = Normal), log id: c196c0c
>2014-07-25 12:39:12,546 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] FINISH, SetVmStatusVDSCommand, log id: c196c0c
>2014-07-25 12:39:12,549 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] Lock freed to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM
>, sharedLocks= ]
>2014-07-25 12:39:12,558 INFO  [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-49) [1fd9302b] Running command: RemoveAllVmImagesCommand internal: true. Entities affected :  ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM
>2014-07-25 12:39:12,561 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] Running command: RemoveImageCommand internal: true. Entities affected :  ID: 00000000-0000-0000-0000-000000000000 Type: Storage
>2014-07-25 12:39:12,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, postZeros = false, forceDelete = false), log id: 18912c0e
>2014-07-25 12:39:14,411 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-07-25 12:39:14,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] FINISH, DeleteImageGroupVDSCommand, log id: 68836761
>2014-07-25 12:39:14,993 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-11) [7a73c920] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command e646e1a1-1d54-48ac-801e-b5ab11ea7c27
>2014-07-25 12:39:14,994 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-11) [7a73c920] CommandMultiAsyncTasks::AttachTask: Attaching task 3ede13e8-c87d-4709-adef-0e890e382020 to command e646e1a1-1d54-48ac-801e-b5ab11ea7c27.
>2014-07-25 12:39:15,053 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-50) SPMAsyncTask::PollTask: Polling task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:39:15,059 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-50) BaseAsyncTask::OnTaskEndSuccess: Task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:39:15,061 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-50) CommandAsyncTask::EndActionIfNecessary: All tasks of command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a has ended -> executing endAction
>2014-07-25 12:39:15,062 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-50) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 199afc57-2e3a-4827-9da2-48aaf3f9cb5a): calling endAction .
>2014-07-25 12:39:15,063 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Adding task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:39:15,063 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0
>2014-07-25 12:39:15,086 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-15) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand
>2014-07-25 12:39:15,088 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result.
>2014-07-25 12:39:15,090 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks.
>2014-07-25 12:39:15,095 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-15) SPMAsyncTask::ClearAsyncTask: Attempting to clear task db8a1b94-2090-4c17-b6ba-f686f03e2b01
>2014-07-25 12:39:15,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = db8a1b94-2090-4c17-b6ba-f686f03e2b01), log id: 65ec8115
>2014-07-25 12:39:15,116 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Correlation ID: 7eafb87d, Job ID: ab47d22c-85f3-4539-9e4b-ecba3c57dcc1, Call Stack: null, Custom Event ID: -1, Message: Removal of Template blank-disk was initiated by admin.
>2014-07-25 12:39:15,117 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-11) [7a73c920] BaseAsyncTask::startPollingTask: Starting to poll task 3ede13e8-c87d-4709-adef-0e890e382020.
>2014-07-25 12:39:25,067 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-77) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now
>2014-07-25 12:39:28,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] FINISH, DeleteImageGroupVDSCommand, log id: 18912c0e
>2014-07-25 12:39:28,540 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-77) SPMAsyncTask::PollTask: Polling task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:39:28,548 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) [5a453713] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command f1e78c90-a974-4bef-a294-edc2d7b21549
>2014-07-25 12:39:28,549 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-49) [5a453713] CommandMultiAsyncTasks::AttachTask: Attaching task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 to command f1e78c90-a974-4bef-a294-edc2d7b21549.
>2014-07-25 12:39:28,551 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-77) BaseAsyncTask::OnTaskEndSuccess: Task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:39:28,563 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-77) CommandAsyncTask::EndActionIfNecessary: All tasks of command e646e1a1-1d54-48ac-801e-b5ab11ea7c27 has ended -> executing endAction
>2014-07-25 12:39:28,565 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-77) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: e646e1a1-1d54-48ac-801e-b5ab11ea7c27): calling endAction .
>2014-07-25 12:39:28,566 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVmTemplate, executionIndex: 0
>2014-07-25 12:39:28,586 INFO  [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-29) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmTemplateCommand
>2014-07-25 12:39:28,589 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [5a453713] Adding task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 12:39:28,605 INFO  [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-29) Lock freed to object EngineLock [exclusiveLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE
>, sharedLocks= ]
>2014-07-25 12:39:28,629 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) Correlation ID: 7eafb87d, Call Stack: null, Custom Event ID: -1, Message: Removal of Template blank-disk has been completed.
>2014-07-25 12:39:28,631 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVmTemplate completed, handling the result.
>2014-07-25 12:39:28,632 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVmTemplate succeeded, clearing tasks.
>2014-07-25 12:39:28,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=db8a1b94-2090-4c17-b6ba-f686f03e2b01), log id: 1666fcb0
>2014-07-25 12:39:28,641 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-29) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 3ede13e8-c87d-4709-adef-0e890e382020
>2014-07-25 12:39:28,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 3ede13e8-c87d-4709-adef-0e890e382020), log id: 11b62e66
>2014-07-25 12:39:28,706 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [5a453713] Correlation ID: 79983e87, Job ID: 29489b48-6f2a-444b-90fe-0f2995907c5d, Call Stack: null, Custom Event ID: -1, Message: VM runit was successfully removed.
>2014-07-25 12:39:28,707 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) [5a453713] BaseAsyncTask::startPollingTask: Starting to poll task c863b58a-c89b-4663-b2aa-2af3a97ad2c2.
>2014-07-25 12:39:37,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, HSMClearTaskVDSCommand, log id: 1666fcb0
>2014-07-25 12:39:37,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, SPMClearTaskVDSCommand, log id: 65ec8115
>2014-07-25 12:39:37,514 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=3ede13e8-c87d-4709-adef-0e890e382020), log id: 1767915c
>2014-07-25 12:39:37,523 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-15) BaseAsyncTask::RemoveTaskFromDB: Removed task db8a1b94-2090-4c17-b6ba-f686f03e2b01 from DataBase
>2014-07-25 12:39:37,524 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 199afc57-2e3a-4827-9da2-48aaf3f9cb5a
>2014-07-25 12:39:37,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, HSMClearTaskVDSCommand, log id: 1767915c
>2014-07-25 12:39:37,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, SPMClearTaskVDSCommand, log id: 11b62e66
>2014-07-25 12:39:37,653 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-29) BaseAsyncTask::RemoveTaskFromDB: Removed task 3ede13e8-c87d-4709-adef-0e890e382020 from DataBase
>2014-07-25 12:39:37,654 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity e646e1a1-1d54-48ac-801e-b5ab11ea7c27
>2014-07-25 12:39:38,567 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-3) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now
>2014-07-25 12:39:38,575 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-3) SPMAsyncTask::PollTask: Polling task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 12:39:38,581 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-3) BaseAsyncTask::OnTaskEndSuccess: Task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 12:39:38,583 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-3) CommandAsyncTask::EndActionIfNecessary: All tasks of command f1e78c90-a974-4bef-a294-edc2d7b21549 has ended -> executing endAction
>2014-07-25 12:39:38,584 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-3) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: f1e78c90-a974-4bef-a294-edc2d7b21549): calling endAction .
>2014-07-25 12:39:38,585 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0
>2014-07-25 12:39:38,598 INFO  [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-46) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand
>2014-07-25 12:39:38,601 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result.
>2014-07-25 12:39:38,602 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks.
>2014-07-25 12:39:38,607 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-46) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c863b58a-c89b-4663-b2aa-2af3a97ad2c2
>2014-07-25 12:39:38,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = c863b58a-c89b-4663-b2aa-2af3a97ad2c2), log id: 199864ee
>2014-07-25 12:39:38,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c863b58a-c89b-4663-b2aa-2af3a97ad2c2), log id: 640a5262
>2014-07-25 12:39:38,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMClearTaskVDSCommand, log id: 640a5262
>2014-07-25 12:39:38,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMClearTaskVDSCommand, log id: 199864ee
>2014-07-25 12:39:38,713 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-46) BaseAsyncTask::RemoveTaskFromDB: Removed task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 from DataBase
>2014-07-25 12:39:38,714 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity f1e78c90-a974-4bef-a294-edc2d7b21549
>2014-07-25 12:40:24,237 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-82) Setting new tasks map. The map contains now 2 tasks
>2014-07-25 12:40:54,238 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Setting new tasks map. The map contains now 0 tasks
>2014-07-25 12:40:54,239 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-25 13:21:50,014 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginAdminUserCommand internal: false.
>2014-07-25 13:21:50,020 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-25 13:21:50,119 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-07-25 13:23:54,281 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized
>2014-07-25 13:23:54,579 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf.
>2014-07-25 13:23:54,814 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [18e37f83] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 13:23:54,824 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [18e37f83] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:54,844 INFO  [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4168ff2b] Running command: CreateOvfStoresForStorageDomainCommand internal: true.
>2014-07-25 13:23:54,848 INFO  [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4e0946ad] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:54,865 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-13) [af148f1] Running command: AddDiskCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:54,887 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:54,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21, newImageDescription = OVF_STORE), log id: 7cdb2fb7
>2014-07-25 13:23:54,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-25 13:23:55,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] FINISH, CreateImageVDSCommand, return: 1f5f82d1-85c2-4386-8764-02bab6dcbc21, log id: 7cdb2fb7
>2014-07-25 13:23:55,262 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 8c80d80c-39bb-4109-a465-c2cb1c952b15
>2014-07-25 13:23:55,263 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] CommandMultiAsyncTasks::AttachTask: Attaching task fc24cd46-4312-49e7-adda-5903f75e3e77 to command 8c80d80c-39bb-4109-a465-c2cb1c952b15.
>2014-07-25 13:23:55,271 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Adding task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 13:23:55,279 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Correlation ID: af148f1, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by Non interactive user.
>2014-07-25 13:23:55,292 INFO  [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [59d5d808] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:55,303 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-13) [665a234b] Running command: AddDiskCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:55,310 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:23:55,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 8596469e-37df-4d9e-9793-d238238131f5, newImageDescription = OVF_STORE), log id: 3021149c
>2014-07-25 13:23:55,325 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-25 13:23:55,751 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] FINISH, CreateImageVDSCommand, return: 8596469e-37df-4d9e-9793-d238238131f5, log id: 3021149c
>2014-07-25 13:23:55,754 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-13) [4dca0641] CommandMultiAsyncTasks::AttachTask: Attaching task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f to command 8c80d80c-39bb-4109-a465-c2cb1c952b15.
>2014-07-25 13:23:55,762 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) [4dca0641] Adding task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 13:23:55,771 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4dca0641] Correlation ID: 665a234b, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by Non interactive user.
>2014-07-25 13:23:55,774 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) [4dca0641] BaseAsyncTask::startPollingTask: Starting to poll task fc24cd46-4312-49e7-adda-5903f75e3e77.
>2014-07-25 13:23:55,775 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) [4dca0641] BaseAsyncTask::startPollingTask: Starting to poll task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f.
>2014-07-25 13:23:55,776 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 13:23:58,692 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-27) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now
>2014-07-25 13:23:58,698 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-27) SPMAsyncTask::PollTask: Polling task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 13:23:58,699 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-27) SPMAsyncTask::PollTask: Polling task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-07-25 13:23:58,701 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-27) Finished polling Tasks, will poll again in 10 seconds.
>2014-07-25 13:24:08,708 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) SPMAsyncTask::PollTask: Polling task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 13:24:08,710 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) BaseAsyncTask::OnTaskEndSuccess: Task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 13:24:08,713 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-37) Task with DB Task ID 946ebb38-af59-4f05-9376-2dc47abe3f46 and VDSM Task ID fc24cd46-4312-49e7-adda-5903f75e3e77 is in state Polling. End action for command 8c80d80c-39bb-4109-a465-c2cb1c952b15 will proceed when all the entitys tasks are completed.
>2014-07-25 13:24:08,715 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) SPMAsyncTask::PollTask: Polling task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 13:24:08,718 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) BaseAsyncTask::OnTaskEndSuccess: Task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 13:24:08,721 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-37) CommandAsyncTask::EndActionIfNecessary: All tasks of command 8c80d80c-39bb-4109-a465-c2cb1c952b15 has ended -> executing endAction
>2014-07-25 13:24:08,722 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-37) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 8c80d80c-39bb-4109-a465-c2cb1c952b15): calling endAction .
>2014-07-25 13:24:08,724 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction CreateOvfStoresForStorageDomain, executionIndex: 0
>2014-07-25 13:24:08,726 INFO  [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] Ending command successfully: org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand
>2014-07-25 13:24:08,739 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-25 13:24:08,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 1a694bb9
>2014-07-25 13:24:08,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@c25fab30, log id: 1a694bb9
>2014-07-25 13:24:08,794 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-25 13:24:08,797 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 6c3de6b5
>2014-07-25 13:24:08,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@67a79d02, log id: 6c3de6b5
>2014-07-25 13:24:08,871 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:24:08,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 4e546ca7
>2014-07-25 13:24:08,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 13:24:08,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":null}
>
>2014-07-25 13:24:09,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] FINISH, SetVolumeDescriptionVDSCommand, log id: 4e546ca7
>2014-07-25 13:24:09,127 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION
>]
>2014-07-25 13:24:09,131 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:24:09,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3afb9ecf
>2014-07-25 13:24:09,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeVdsBrokerCommand: 
>2014-07-25 13:24:09,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                size=10,240
>
>2014-07-25 13:24:09,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, UploadStreamVDSCommand, log id: 3afb9ecf
>2014-07-25 13:24:10,012 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [30258eda] CommandMultiAsyncTasks::AttachTask: Attaching task b1a6bc7b-1427-405f-8d32-41e4358fd139 to command 8c80d80c-39bb-4109-a465-c2cb1c952b15.
>2014-07-25 13:24:10,022 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [30258eda] Adding task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 13:24:10,024 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION
>]
>2014-07-25 13:24:10,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 577c34e1
>2014-07-25 13:24:10,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 13:24:10,038 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014","Size":10240}
>
>2014-07-25 13:24:10,340 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, SetVolumeDescriptionVDSCommand, log id: 577c34e1
>2014-07-25 13:24:10,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 488ca852
>2014-07-25 13:24:10,352 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 13:24:10,353 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":null}
>
>2014-07-25 13:24:10,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, SetVolumeDescriptionVDSCommand, log id: 488ca852
>2014-07-25 13:24:10,539 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION
>]
>2014-07-25 13:24:10,542 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:24:10,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 73547d41
>2014-07-25 13:24:10,545 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- executeVdsBrokerCommand: 
>2014-07-25 13:24:10,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                size=10,240
>
>2014-07-25 13:24:11,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, UploadStreamVDSCommand, log id: 73547d41
>2014-07-25 13:24:11,236 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandMultiAsyncTasks::AttachTask: Attaching task bbc204bf-5244-41ff-bf18-1de1ce9966ba to command 8c80d80c-39bb-4109-a465-c2cb1c952b15.
>2014-07-25 13:24:11,243 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Adding task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 13:24:11,245 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION
>]
>2014-07-25 13:24:11,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 40b7a99
>2014-07-25 13:24:11,252 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 13:24:11,253 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014","Size":10240}
>
>2014-07-25 13:24:11,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SetVolumeDescriptionVDSCommand, log id: 40b7a99
>2014-07-25 13:24:11,457 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::startPollingTask: Starting to poll task b1a6bc7b-1427-405f-8d32-41e4358fd139.
>2014-07-25 13:24:11,458 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::startPollingTask: Starting to poll task bbc204bf-5244-41ff-bf18-1de1ce9966ba.
>2014-07-25 13:24:11,459 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain completed, handling the result.
>2014-07-25 13:24:11,460 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain succeeded, clearing tasks.
>2014-07-25 13:24:11,462 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task fc24cd46-4312-49e7-adda-5903f75e3e77
>2014-07-25 13:24:11,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = fc24cd46-4312-49e7-adda-5903f75e3e77), log id: 73f70cc7
>2014-07-25 13:24:11,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fc24cd46-4312-49e7-adda-5903f75e3e77), log id: 6068184b
>2014-07-25 13:24:11,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, HSMClearTaskVDSCommand, log id: 6068184b
>2014-07-25 13:24:11,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SPMClearTaskVDSCommand, log id: 73f70cc7
>2014-07-25 13:24:11,613 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::RemoveTaskFromDB: Removed task fc24cd46-4312-49e7-adda-5903f75e3e77 from DataBase
>2014-07-25 13:24:11,614 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f
>2014-07-25 13:24:11,616 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f), log id: 3bfddcbc
>2014-07-25 13:24:11,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f), log id: 17a6d969
>2014-07-25 13:24:11,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, HSMClearTaskVDSCommand, log id: 17a6d969
>2014-07-25 13:24:11,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SPMClearTaskVDSCommand, log id: 3bfddcbc
>2014-07-25 13:24:11,714 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::RemoveTaskFromDB: Removed task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f from DataBase
>2014-07-25 13:24:11,716 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] [within thread]: Some of the tasks related to command id 8c80d80c-39bb-4109-a465-c2cb1c952b15 were not cleared yet (Task id bbc204bf-5244-41ff-bf18-1de1ce9966ba is in state Polling).
>2014-07-25 13:24:18,725 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-64) Polling and updating Async Tasks: 4 tasks, 2 tasks to poll now
>2014-07-25 13:24:18,731 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) SPMAsyncTask::PollTask: Polling task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 13:24:18,733 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) BaseAsyncTask::OnTaskEndSuccess: Task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 13:24:18,734 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-64) Task with DB Task ID 3aad0be1-afd6-4131-8135-7425d953b511 and VDSM Task ID bbc204bf-5244-41ff-bf18-1de1ce9966ba is in state Polling. End action for command 8c80d80c-39bb-4109-a465-c2cb1c952b15 will proceed when all the entitys tasks are completed.
>2014-07-25 13:24:18,736 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) SPMAsyncTask::PollTask: Polling task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 13:24:18,737 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) BaseAsyncTask::OnTaskEndSuccess: Task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 13:24:18,739 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-64) CommandAsyncTask::EndActionIfNecessary: All tasks of command 8c80d80c-39bb-4109-a465-c2cb1c952b15 has ended -> executing endAction
>2014-07-25 13:24:18,740 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-64) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 8c80d80c-39bb-4109-a465-c2cb1c952b15): calling endAction .
>2014-07-25 13:24:18,741 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0
>2014-07-25 13:24:18,771 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand
>2014-07-25 13:24:18,773 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result.
>2014-07-25 13:24:18,774 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks.
>2014-07-25 13:24:18,775 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task bbc204bf-5244-41ff-bf18-1de1ce9966ba
>2014-07-25 13:24:18,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = bbc204bf-5244-41ff-bf18-1de1ce9966ba), log id: 602cc445
>2014-07-25 13:24:18,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=bbc204bf-5244-41ff-bf18-1de1ce9966ba), log id: 1b09bf57
>2014-07-25 13:24:18,813 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, HSMClearTaskVDSCommand, log id: 1b09bf57
>2014-07-25 13:24:18,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, SPMClearTaskVDSCommand, log id: 602cc445
>2014-07-25 13:24:18,832 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] BaseAsyncTask::RemoveTaskFromDB: Removed task bbc204bf-5244-41ff-bf18-1de1ce9966ba from DataBase
>2014-07-25 13:24:18,833 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task b1a6bc7b-1427-405f-8d32-41e4358fd139
>2014-07-25 13:24:18,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = b1a6bc7b-1427-405f-8d32-41e4358fd139), log id: 7b857e7b
>2014-07-25 13:24:18,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b1a6bc7b-1427-405f-8d32-41e4358fd139), log id: 7932fc97
>2014-07-25 13:24:18,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, HSMClearTaskVDSCommand, log id: 7932fc97
>2014-07-25 13:24:18,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, SPMClearTaskVDSCommand, log id: 7b857e7b
>2014-07-25 13:24:18,924 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] BaseAsyncTask::RemoveTaskFromDB: Removed task b1a6bc7b-1427-405f-8d32-41e4358fd139 from DataBase
>2014-07-25 13:24:18,926 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 8c80d80c-39bb-4109-a465-c2cb1c952b15
>2014-07-25 13:25:24,278 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-18) Setting new tasks map. The map contains now 0 tasks
>2014-07-25 13:25:24,279 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-18) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-25 13:34:11,880 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Lock Acquired to object EngineLock [exclusiveLocks= key: testtesttest value: VM_NAME
>, sharedLocks= ]
>2014-07-25 13:34:11,932 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Running command: AddVmFromScratchCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-25 13:34:12,053 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [76d45966] Correlation ID: 76d45966, Job ID: 24d01812-2438-4a5f-b1f9-5e96401f78f9, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was created by admin.
>2014-07-25 13:34:12,068 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Lock freed to object EngineLock [exclusiveLocks= key: testtesttest value: VM_NAME
>, sharedLocks= ]
>2014-07-25 13:34:21,162 INFO  [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-10) [d9e7c59] Running command: AddVmInterfaceCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM,  ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile
>2014-07-25 13:34:21,211 INFO  [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-10) [3a8e225] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:21,262 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3a8e225] Correlation ID: 3a8e225, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM testtesttest. (User: admin)
>2014-07-25 13:34:21,273 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3a8e225] Correlation ID: d9e7c59, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM testtesttest. (User: admin)
>2014-07-25 13:34:21,697 INFO  [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-7) [5e069bea] Running command: ReorderVmNicsCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:26,154 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [728fbda2] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM_DISK_BOOT
>, sharedLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>]
>2014-07-25 13:34:26,217 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [728fbda2] Running command: AddDiskCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:34:26,259 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [43def421] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 13:34:26,274 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [43def421] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM_DISK_BOOT
>, sharedLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>]
>2014-07-25 13:34:26,276 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageSizeInBytes = 21474836480, volumeFormat = RAW, newImageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, newImageDescription = ), log id: 7d691e51
>2014-07-25 13:34:26,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-25 13:34:26,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] FINISH, CreateImageVDSCommand, return: b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, log id: 7d691e51
>2014-07-25 13:34:26,671 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-2) [43def421] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 0500fc29-af6c-44ff-be85-2f60df4cf00a
>2014-07-25 13:34:26,672 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-2) [43def421] CommandMultiAsyncTasks::AttachTask: Attaching task cbda6151-cac6-46ea-af0c-206457c6cc2b to command 0500fc29-af6c-44ff-be85-2f60df4cf00a.
>2014-07-25 13:34:26,688 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-2) [43def421] Adding task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 13:34:26,717 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [43def421] Correlation ID: 728fbda2, Job ID: fba17df4-f9d0-4038-aa71-316111e063ca, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of testtesttest_Disk1 was initiated on VM testtesttest by admin.
>2014-07-25 13:34:26,719 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-2) [43def421] BaseAsyncTask::startPollingTask: Starting to poll task cbda6151-cac6-46ea-af0c-206457c6cc2b.
>2014-07-25 13:34:28,763 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-52) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-07-25 13:34:28,770 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) SPMAsyncTask::PollTask: Polling task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 13:34:28,776 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) BaseAsyncTask::OnTaskEndSuccess: Task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 13:34:28,778 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-52) CommandAsyncTask::EndActionIfNecessary: All tasks of command 0500fc29-af6c-44ff-be85-2f60df4cf00a has ended -> executing endAction
>2014-07-25 13:34:28,779 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-52) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 0500fc29-af6c-44ff-be85-2f60df4cf00a): calling endAction .
>2014-07-25 13:34:28,780 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0
>2014-07-25 13:34:28,791 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-49) [728fbda2] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand
>2014-07-25 13:34:28,798 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-25 13:34:28,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b), log id: 3f76efbe
>2014-07-25 13:34:28,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@fd653795, log id: 3f76efbe
>2014-07-25 13:34:28,855 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 728fbda2, Call Stack: null, Custom Event ID: -1, Message: The disk testtesttest_Disk1 was successfully added to VM testtesttest.
>2014-07-25 13:34:28,856 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result.
>2014-07-25 13:34:28,857 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks.
>2014-07-25 13:34:28,862 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) SPMAsyncTask::ClearAsyncTask: Attempting to clear task cbda6151-cac6-46ea-af0c-206457c6cc2b
>2014-07-25 13:34:28,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = cbda6151-cac6-46ea-af0c-206457c6cc2b), log id: 6fdb8433
>2014-07-25 13:34:28,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cbda6151-cac6-46ea-af0c-206457c6cc2b), log id: 34c469a4
>2014-07-25 13:34:28,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, HSMClearTaskVDSCommand, log id: 34c469a4
>2014-07-25 13:34:28,917 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, SPMClearTaskVDSCommand, log id: 6fdb8433
>2014-07-25 13:34:28,928 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) BaseAsyncTask::RemoveTaskFromDB: Removed task cbda6151-cac6-46ea-af0c-206457c6cc2b from DataBase
>2014-07-25 13:34:28,930 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 0500fc29-af6c-44ff-be85-2f60df4cf00a
>2014-07-25 13:34:34,885 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] Lock Acquired to object EngineLock [exclusiveLocks= key: a2660663-ef34-41ee-a580-0fc425183c36 value: VM
>, sharedLocks= ]
>2014-07-25 13:34:34,890 WARN  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_CANNOT_RUN_ACTION_ON_NON_MANAGED_VM
>2014-07-25 13:34:34,891 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] Lock freed to object EngineLock [exclusiveLocks= key: a2660663-ef34-41ee-a580-0fc425183c36 value: VM
>, sharedLocks= ]
>2014-07-25 13:34:38,784 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-25 13:34:38,821 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: 39727cec
>2014-07-25 13:34:38,822 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 39727cec
>2014-07-25 13:34:38,888 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Running command: RunVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:38,940 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@f5735a74), log id: 65b1b34f
>2014-07-25 13:34:38,946 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, UpdateVmDynamicDataVDSCommand, log id: 65b1b34f
>2014-07-25 13:34:38,960 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 32bc339c
>2014-07-25 13:34:38,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 4df366f
>2014-07-25 13:34:39,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs
>CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog
>ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0
>Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiODM3M2I1ZmUtZjllYS00Y2Q4
>LTk0MjQtMDdmNmE5MjQzMjdjIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz
>ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ==
>, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg
>YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp
>cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0
>Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg
>IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK
>, openstack/content/0000=YXV0bwo=
>}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=407cc02a-1dee-4993-a2a1-5ef7a2e1ee34}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, index=0, iface=virtio, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true
>2014-07-25 13:34:39,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, CreateVmFromCloudInitVDSCommand, log id: 4df366f
>2014-07-25 13:34:39,064 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 32bc339c
>2014-07-25 13:34:39,065 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-25 13:34:39,072 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Correlation ID: 4f3adf5f, Job ID: 015cfb40-a61b-43a0-9793-637c0531876a, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack).
>2014-07-25 13:34:41,386 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp
>2014-07-25 13:34:41,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-78) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 5e2e6401
>2014-07-25 13:34:41,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22150, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@1769b96b, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 5e2e6401
>2014-07-25 13:34:41,411 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-25 13:34:41,413 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-25 13:34:43,248 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-2) [7314faa8] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:43,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-2) [7314faa8] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=Z9uKew2SVSyz, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 76a8d70
>2014-07-25 13:34:43,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-2) [7314faa8] FINISH, SetVmTicketVDSCommand, log id: 76a8d70
>2014-07-25 13:34:43,320 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [7314faa8] Correlation ID: 7314faa8, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:34:51,728 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-5) [8da3838] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:51,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-5) [8da3838] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=WUUWB6jV/ajY, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 1149e50c
>2014-07-25 13:34:51,775 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-5) [8da3838] FINISH, SetVmTicketVDSCommand, log id: 1149e50c
>2014-07-25 13:34:51,785 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [8da3838] Correlation ID: 8da3838, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:34:56,381 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-8) [7f03b371] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:34:56,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [7f03b371] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=gdVxAfkMKBLi, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6a44ef1a
>2014-07-25 13:34:56,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [7f03b371] FINISH, SetVmTicketVDSCommand, log id: 6a44ef1a
>2014-07-25 13:34:56,435 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [7f03b371] Correlation ID: 7f03b371, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:35:01,904 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [5f068865] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:35:01,909 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [5f068865] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=OzD3NSenMHYU, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 5ae021e2
>2014-07-25 13:35:01,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [5f068865] FINISH, SetVmTicketVDSCommand, log id: 5ae021e2
>2014-07-25 13:35:01,968 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f068865] Correlation ID: 5f068865, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:35:03,925 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-1) [136ab59] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:35:03,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-1) [136ab59] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=wZzwEsPwNpgh, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 2ae4a6b5
>2014-07-25 13:35:03,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-1) [136ab59] FINISH, SetVmTicketVDSCommand, log id: 2ae4a6b5
>2014-07-25 13:35:03,996 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [136ab59] Correlation ID: 136ab59, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:35:23,004 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:35:23,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=kq2FkK7R8E9w, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 3cfbc3f7
>2014-07-25 13:35:23,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] FINISH, SetVmTicketVDSCommand, log id: 3cfbc3f7
>2014-07-25 13:35:23,064 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [32d0d16b] Correlation ID: 32d0d16b, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:35:26,815 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:35:26,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=h9B8Ex/9G2vq, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 1c20ceae
>2014-07-25 13:35:26,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] FINISH, SetVmTicketVDSCommand, log id: 1c20ceae
>2014-07-25 13:35:26,884 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [fd5ef46] Correlation ID: fd5ef46, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:35:39,207 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from PoweringUp --> Up
>2014-07-25 13:35:39,222 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: 4f3adf5f, Job ID: 015cfb40-a61b-43a0-9793-637c0531876a, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack
>2014-07-25 13:35:54,288 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Setting new tasks map. The map contains now 0 tasks
>2014-07-25 13:35:54,289 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-25 13:35:59,997 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [6426061] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:36:00,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [6426061] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=ZVhSbBuWzn64, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7d89e0f3
>2014-07-25 13:36:00,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [6426061] FINISH, SetVmTicketVDSCommand, log id: 7d89e0f3
>2014-07-25 13:36:00,058 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6426061] Correlation ID: 6426061, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:36:18,367 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] Running command: StopVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:36:18,371 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 428a7c40
>2014-07-25 13:36:18,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 44ca626f
>2014-07-25 13:36:18,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] FINISH, DestroyVDSCommand, log id: 44ca626f
>2014-07-25 13:36:18,811 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] FINISH, DestroyVmVDSCommand, return: Down, log id: 428a7c40
>2014-07-25 13:36:18,825 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-49) [783724f9] Running command: ProcessDownVmCommand internal: true.
>2014-07-25 13:36:18,834 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] Correlation ID: 3149e2e5, Job ID: 1a89d746-1ce6-486c-b832-eb3fae4abfd3, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest powered off by admin (Host: rack) (Reason: Not Specified).
>2014-07-25 13:36:21,033 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-25 13:36:21,115 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: 118a0d1f
>2014-07-25 13:36:21,117 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 118a0d1f
>2014-07-25 13:36:21,156 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Running command: RunVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:36:21,193 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@4dd29a5c), log id: 12e298f8
>2014-07-25 13:36:21,197 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, UpdateVmDynamicDataVDSCommand, log id: 12e298f8
>2014-07-25 13:36:21,203 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 5c4e97ea
>2014-07-25 13:36:21,208 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 6c32b26a
>2014-07-25 13:36:21,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, index=0, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true
>2014-07-25 13:36:21,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, CreateVDSCommand, log id: 6c32b26a
>2014-07-25 13:36:21,271 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5c4e97ea
>2014-07-25 13:36:21,272 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-25 13:36:21,278 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Correlation ID: 2b8e1554, Job ID: 3cbd46e7-5428-4259-90cf-b4663f92e4bc, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack).
>2014-07-25 13:36:24,901 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp
>2014-07-25 13:36:24,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-8) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 4da758dd
>2014-07-25 13:36:24,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-8) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22477, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@25bb3cff, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 4da758dd
>2014-07-25 13:36:24,921 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-25 13:36:24,923 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-25 13:36:26,878 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-25 13:36:26,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=dYXS3cMpK8FZ, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7bd1bc81
>2014-07-25 13:36:27,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] FINISH, SetVmTicketVDSCommand, log id: 7bd1bc81
>2014-07-25 13:36:27,033 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4dbf49a2] Correlation ID: 4dbf49a2, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest
>2014-07-25 13:36:31,070 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM testtesttest.
>2014-07-25 13:37:22,671 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from PoweringUp --> Up
>2014-07-25 13:37:22,686 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: 2b8e1554, Job ID: 3cbd46e7-5428-4259-90cf-b4663f92e4bc, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack
>2014-07-25 13:46:36,595 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Up --> RebootInProgress
>2014-07-25 13:46:36,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-88) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 6b7c285
>2014-07-25 13:46:36,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22477, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=10.34.130.128, devices=[Ljava.lang.Object;@a43ad35, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 6b7c285
>2014-07-25 13:46:36,619 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-25 13:46:36,621 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-25 13:47:34,316 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from RebootInProgress --> Up
>2014-07-25 14:09:34,257 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-07-25 14:23:54,290 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized
>2014-07-25 14:23:55,778 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf.
>2014-07-25 14:23:55,908 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 14:23:55,912 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 14:23:55,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 57d4eb14
>2014-07-25 14:23:55,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 14:23:55,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014"}
>
>2014-07-25 14:23:56,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] FINISH, SetVolumeDescriptionVDSCommand, log id: 57d4eb14
>2014-07-25 14:23:56,099 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 14:23:56,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7d9ee577
>2014-07-25 14:23:56,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeVdsBrokerCommand: 
>2014-07-25 14:23:56,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                size=10,240
>
>2014-07-25 14:23:57,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, UploadStreamVDSCommand, log id: 7d9ee577
>2014-07-25 14:23:57,649 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-58) [5e092991] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 09ff096c-0935-490c-b787-df4037d31160
>2014-07-25 14:23:57,650 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-58) [5e092991] CommandMultiAsyncTasks::AttachTask: Attaching task e59dd314-6a4d-43c0-b873-8403d1791fb9 to command 09ff096c-0935-490c-b787-df4037d31160.
>2014-07-25 14:23:57,657 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) [5e092991] Adding task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 14:23:57,658 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 14:23:57,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 707eb765
>2014-07-25 14:23:57,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 14:23:57,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014","Size":10240}
>
>2014-07-25 14:23:57,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, SetVolumeDescriptionVDSCommand, log id: 707eb765
>2014-07-25 14:23:57,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 25645b8c
>2014-07-25 14:23:57,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 14:23:57,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014"}
>
>2014-07-25 14:23:58,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, SetVolumeDescriptionVDSCommand, log id: 25645b8c
>2014-07-25 14:23:58,080 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 14:23:58,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16e0ce98
>2014-07-25 14:23:58,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- executeVdsBrokerCommand: 
>2014-07-25 14:23:58,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                size=10,240
>
>2014-07-25 14:23:58,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, UploadStreamVDSCommand, log id: 16e0ce98
>2014-07-25 14:23:58,875 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-58) [266916e1] CommandMultiAsyncTasks::AttachTask: Attaching task 877232e6-8358-498a-8334-857f5e5748fb to command 09ff096c-0935-490c-b787-df4037d31160.
>2014-07-25 14:23:58,882 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Adding task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-25 14:23:58,884 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-07-25 14:23:58,885 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-07-25 14:23:58,886 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-07-25 14:23:58,887 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-07-25 14:23:58,888 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 14:23:58,891 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 46da6ba3
>2014-07-25 14:23:58,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-25 14:23:58,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014","Size":10240}
>
>2014-07-25 14:23:59,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, SetVolumeDescriptionVDSCommand, log id: 46da6ba3
>2014-07-25 14:23:59,107 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) [266916e1] BaseAsyncTask::startPollingTask: Starting to poll task e59dd314-6a4d-43c0-b873-8403d1791fb9.
>2014-07-25 14:23:59,109 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) [266916e1] BaseAsyncTask::startPollingTask: Starting to poll task 877232e6-8358-498a-8334-857f5e5748fb.
>2014-07-25 14:24:08,915 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-80) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now
>2014-07-25 14:24:08,922 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) SPMAsyncTask::PollTask: Polling task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 14:24:08,923 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) BaseAsyncTask::OnTaskEndSuccess: Task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 14:24:08,926 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-80) Task with DB Task ID 750b96d9-f392-4006-8f82-aedd720e661f and VDSM Task ID 877232e6-8358-498a-8334-857f5e5748fb is in state Polling. End action for command 09ff096c-0935-490c-b787-df4037d31160 will proceed when all the entitys tasks are completed.
>2014-07-25 14:24:08,928 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) SPMAsyncTask::PollTask: Polling task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-25 14:24:08,930 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) BaseAsyncTask::OnTaskEndSuccess: Task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-25 14:24:08,932 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-80) CommandAsyncTask::EndActionIfNecessary: All tasks of command 09ff096c-0935-490c-b787-df4037d31160 has ended -> executing endAction
>2014-07-25 14:24:08,933 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-80) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 09ff096c-0935-490c-b787-df4037d31160): calling endAction .
>2014-07-25 14:24:08,935 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0
>2014-07-25 14:24:08,978 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand
>2014-07-25 14:24:08,979 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-07-25 14:24:08,981 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-07-25 14:24:08,982 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-07-25 14:24:08,983 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-07-25 14:24:08,984 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-25 14:24:08,987 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result.
>2014-07-25 14:24:08,988 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks.
>2014-07-25 14:24:08,990 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 877232e6-8358-498a-8334-857f5e5748fb
>2014-07-25 14:24:08,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 877232e6-8358-498a-8334-857f5e5748fb), log id: 412a9eb0
>2014-07-25 14:24:08,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=877232e6-8358-498a-8334-857f5e5748fb), log id: 5518563c
>2014-07-25 14:24:09,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, HSMClearTaskVDSCommand, log id: 5518563c
>2014-07-25 14:24:09,042 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, SPMClearTaskVDSCommand, log id: 412a9eb0
>2014-07-25 14:24:09,048 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] BaseAsyncTask::RemoveTaskFromDB: Removed task 877232e6-8358-498a-8334-857f5e5748fb from DataBase
>2014-07-25 14:24:09,049 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task e59dd314-6a4d-43c0-b873-8403d1791fb9
>2014-07-25 14:24:09,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = e59dd314-6a4d-43c0-b873-8403d1791fb9), log id: 31e1683d
>2014-07-25 14:24:09,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=e59dd314-6a4d-43c0-b873-8403d1791fb9), log id: 4365f981
>2014-07-25 14:24:09,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, HSMClearTaskVDSCommand, log id: 4365f981
>2014-07-25 14:24:09,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, SPMClearTaskVDSCommand, log id: 31e1683d
>2014-07-25 14:24:09,174 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] BaseAsyncTask::RemoveTaskFromDB: Removed task e59dd314-6a4d-43c0-b873-8403d1791fb9 from DataBase
>2014-07-25 14:24:09,174 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 09ff096c-0935-490c-b787-df4037d31160
>2014-07-25 14:25:24,331 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-47) Setting new tasks map. The map contains now 0 tasks
>2014-07-25 14:25:24,332 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-47) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-25 15:23:54,297 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized
>2014-07-25 15:23:59,109 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf.
>2014-07-25 15:37:15,729 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-25 16:13:52,364 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM testtesttest.
>2014-07-25 16:23:54,305 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized
>2014-07-25 16:23:59,142 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-07-25 17:15:38,518 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-25 17:23:54,314 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized
>2014-07-25 17:23:59,167 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf.
>2014-07-25 18:20:56,511 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.11967 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-25 18:23:54,322 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized
>2014-07-25 18:23:59,182 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
>2014-07-25 19:23:54,328 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-07-25 19:23:59,194 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf.
>2014-07-25 20:23:54,334 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized
>2014-07-25 20:23:59,224 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-25 21:14:53,157 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-25 21:16:08,810 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Up --> NotResponding
>2014-07-25 21:16:08,885 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest is not responding.
>2014-07-25 21:16:18,066 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-25 21:16:18,071 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 21:16:18,076 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-28) [8ef1638] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-25 21:16:18,094 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [8ef1638] Correlation ID: 8ef1638, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-25 21:18:18,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-25 21:18:18,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-25 21:18:18,110 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-25 21:18:18,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-25 21:18:18,181 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 21:18:18,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 36c0c0a3
>2014-07-25 21:18:18,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 41a9228a
>2014-07-25 21:18:18,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-25 21:20:00,010 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) Autorecovering 1 storage domains
>2014-07-25 21:20:00,012 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-25 21:20:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 21:20:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/25/14 9:20 PM
>2014-07-25 21:20:00,040 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-25 21:20:00,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c60160d
>2014-07-25 21:21:18,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-25 21:21:18,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-25 21:21:18,091 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-25 21:21:18,092 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-25 21:21:18,223 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-07-25 21:21:18,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] FINISH, SpmStopVDSCommand, log id: 41a9228a
>2014-07-25 21:21:18,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] FINISH, ResetIrsVDSCommand, log id: 36c0c0a3
>2014-07-25 21:21:18,248 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [1efdb026] Correlation ID: 1efdb026, Job ID: 94c9f882-4940-4be9-8248-5261de5619db, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-25 21:21:18,256 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [8ef1638] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-25 21:21:18,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-25 21:21:18,320 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-25 21:21:18,366 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-25 21:21:18,397 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Removing vds [] from the domain in maintenance cache
>2014-07-25 21:21:18,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Removing host(s) [] from hosts unseen domain report cache
>2014-07-25 21:21:18,399 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [8ef1638] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-25 21:22:18,099 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-25 21:22:18,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-25 21:22:18,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] FINISH, ConnectStorageServerVDSCommand, log id: 6c60160d
>2014-07-25 21:22:18,103 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host rack is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued.
>2014-07-25 21:22:18,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-25 21:22:18,108 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-25 21:22:18,106 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-25 21:22:18,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-25 21:22:18,125 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-25 21:22:18,112 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-25 21:22:18,126 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/25/14 9:22 PM
>2014-07-25 21:22:18,153 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-25 21:22:21,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 42662e48
>2014-07-25 21:22:21,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GetHardwareInfoVDSCommand, log id: 42662e48
>2014-07-25 21:22:21,517 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [3060a9b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-25 21:22:21,531 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) [3060a9b0] VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from NotResponding --> Paused
>2014-07-25 21:22:21,537 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [e9159fd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 21:22:21,556 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-25 21:22:21,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 195e5835
>2014-07-25 21:22:21,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 195e5835
>2014-07-25 21:22:21,588 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] Host rack storage connection was succeeded 
>2014-07-25 21:22:21,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5cfee7ec
>2014-07-25 21:22:28,120 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-25 21:23:21,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 5cfee7ec
>2014-07-25 21:23:21,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4a847345
>2014-07-25 21:23:22,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4a847345
>2014-07-25 21:23:22,096 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [b23fc58] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-25 21:23:22,104 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [b23fc58] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-25 21:23:22,129 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-25 21:23:28,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-25 21:23:28,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-25 21:23:28,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 788cebfe
>2014-07-25 21:23:28,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling started: taskId = 5d4b0fec-6945-4715-9afd-83382a58244b
>2014-07-25 21:23:29,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling ended: taskId = 5d4b0fec-6945-4715-9afd-83382a58244b task status = finished
>2014-07-25 21:23:29,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling ended, spm status: SPM
>2014-07-25 21:23:29,367 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5d4b0fec-6945-4715-9afd-83382a58244b), log id: 17bf8e5c
>2014-07-25 21:23:29,411 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] FINISH, HSMClearTaskVDSCommand, log id: 17bf8e5c
>2014-07-25 21:23:29,412 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a0a0eba, log id: 788cebfe
>2014-07-25 21:23:29,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-25 21:23:29,422 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [8ef1638] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-25 21:23:29,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 501306e2
>2014-07-25 21:23:29,469 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-25 21:23:29,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42f72653
>2014-07-25 21:23:29,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42f72653
>2014-07-25 21:23:29,479 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 501306e2
>2014-07-25 21:23:29,480 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-33) [8ef1638] Discovered no tasks on Storage Pool Default
>2014-07-25 21:23:54,340 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized
>2014-07-25 21:23:59,251 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-25 22:23:54,345 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [2acc035c] Principal admin::internal synchronized
>2014-07-25 22:23:59,281 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-07-25 23:23:54,355 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-07-25 23:23:59,294 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf.
>2014-07-26 00:23:54,363 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized
>2014-07-26 00:23:59,308 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf.
>2014-07-26 01:23:54,372 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) Principal admin::internal synchronized
>2014-07-26 01:23:59,340 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf.
>2014-07-26 02:23:54,377 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized
>2014-07-26 02:23:59,355 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
>2014-07-26 03:23:54,384 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized
>2014-07-26 03:23:59,479 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf.
>2014-07-26 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-61) Start deleteAgedOutAuditLogs
>2014-07-26 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Start deleteAgedOutCommandEntities
>2014-07-26 03:35:35,049 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-61) Finished deleteAgedOutAuditLogs
>2014-07-26 03:35:35,051 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Finished deleteAgedOutCommandEntities
>2014-07-26 04:23:54,390 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [4dca0641] Principal admin::internal synchronized
>2014-07-26 04:23:59,494 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-07-26 05:23:54,397 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized
>2014-07-26 05:23:59,508 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
>2014-07-26 06:23:54,408 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) Principal admin::internal synchronized
>2014-07-26 06:23:59,524 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf.
>2014-07-26 06:28:41,941 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-26 06:30:13,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 06:30:13,590 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 06:30:13,597 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [2734597c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-26 06:30:13,618 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [2734597c] Correlation ID: 2734597c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-26 06:32:13,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 06:32:13,631 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-26 06:32:13,632 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-26 06:32:13,633 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 06:32:13,668 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-26 06:32:13,694 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 537acf4
>2014-07-26 06:32:13,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6063da94
>2014-07-26 06:32:13,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-26 06:34:13,601 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-26 06:34:13,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] FINISH, SpmStopVDSCommand, log id: 6063da94
>2014-07-26 06:34:13,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-26 06:34:13,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-26 06:34:13,608 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-89) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-26 06:34:13,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] FINISH, ResetIrsVDSCommand, log id: 537acf4
>2014-07-26 06:34:13,611 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-26 06:34:13,613 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-26 06:34:13,634 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] Correlation ID: 5e9a1b3a, Job ID: b98e617d-8439-4ba7-bf5d-74d935be6985, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-26 06:34:13,654 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [2734597c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-26 06:34:13,657 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-26 06:34:13,660 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-26 06:34:13,731 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-26 06:34:13,749 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-26 06:34:13,757 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] Removing vds [] from the domain in maintenance cache
>2014-07-26 06:34:13,758 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] Removing host(s) [] from hosts unseen domain report cache
>2014-07-26 06:34:13,758 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [2734597c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-26 06:34:16,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e6ebdf4
>2014-07-26 06:34:16,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) HostName = rack
>2014-07-26 06:34:16,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-26 06:34:16,790 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-26 06:34:16,799 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-26 06:34:20,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2efd43b
>2014-07-26 06:34:20,086 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, GetHardwareInfoVDSCommand, log id: 2efd43b
>2014-07-26 06:34:20,128 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [43049438] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-26 06:34:20,142 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [43049438] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-26 06:34:20,145 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [222469a6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 06:34:20,154 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 06:34:20,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2025934c
>2014-07-26 06:34:20,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 2025934c
>2014-07-26 06:34:20,220 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] Host rack storage connection was succeeded 
>2014-07-26 06:34:20,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 65f4633d
>2014-07-26 06:34:23,616 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-26 06:35:20,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, ConnectStoragePoolVDSCommand, log id: 65f4633d
>2014-07-26 06:35:20,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 381d3566
>2014-07-26 06:35:20,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] FINISH, SetMOMPolicyParametersVDSCommand, log id: 381d3566
>2014-07-26 06:35:20,954 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [75046dac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-26 06:35:20,960 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [75046dac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-26 06:35:20,975 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-26 06:35:24,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-26 06:35:24,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-26 06:35:24,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6952f735
>2014-07-26 06:35:24,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling started: taskId = adefb6d5-a768-45d8-9dce-959bff0b1479
>2014-07-26 06:35:26,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling ended: taskId = adefb6d5-a768-45d8-9dce-959bff0b1479 task status = finished
>2014-07-26 06:35:26,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling ended, spm status: SPM
>2014-07-26 06:35:26,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-92) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=adefb6d5-a768-45d8-9dce-959bff0b1479), log id: 40a12104
>2014-07-26 06:35:26,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, HSMClearTaskVDSCommand, log id: 40a12104
>2014-07-26 06:35:26,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@574079b, log id: 6952f735
>2014-07-26 06:35:26,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-26 06:35:26,738 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-26 06:35:26,740 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-26 06:35:26,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5c47d77f
>2014-07-26 06:35:26,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-26 06:35:26,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 69e92642
>2014-07-26 06:35:26,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69e92642
>2014-07-26 06:35:26,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5c47d77f
>2014-07-26 06:35:26,790 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-2) Discovered no tasks on Storage Pool Default
>2014-07-26 07:23:54,414 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized
>2014-07-26 07:23:59,559 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
>2014-07-26 08:23:54,419 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized
>2014-07-26 08:23:59,575 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-07-26 09:01:58,120 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.61273 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-26 09:23:54,427 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized
>2014-07-26 09:23:59,591 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf.
>2014-07-26 10:23:54,434 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized
>2014-07-26 10:23:59,608 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf.
>2014-07-26 11:23:54,442 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized
>2014-07-26 11:23:59,636 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf.
>2014-07-26 12:23:54,458 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized
>2014-07-26 12:23:59,654 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf.
>2014-07-26 13:23:54,466 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized
>2014-07-26 13:23:59,673 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf.
>2014-07-26 14:23:54,475 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) Principal admin::internal synchronized
>2014-07-26 14:23:59,705 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf.
>2014-07-26 15:23:54,481 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized
>2014-07-26 15:23:59,720 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf.
>2014-07-26 15:56:16,647 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-26 15:57:54,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 15:57:54,454 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 15:57:54,458 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-26 15:57:54,472 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Correlation ID: 3abdaaa6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-26 15:59:54,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 15:59:54,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-26 15:59:54,536 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-26 15:59:54,537 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-26 15:59:54,582 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-26 15:59:54,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 232547d3
>2014-07-26 15:59:54,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 29c94738
>2014-07-26 15:59:54,611 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-26 16:00:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering 1 storage domains
>2014-07-26 16:00:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-26 16:00:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-26 16:00:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/26/14 4:00 PM
>2014-07-26 16:00:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-26 16:00:00,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 59fe87bd
>2014-07-26 16:01:00,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 59fe87bd
>2014-07-26 16:01:00,039 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/26/14 4:01 PM
>2014-07-26 16:01:54,501 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-26 16:01:54,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-26 16:01:54,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-26 16:01:54,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-26 16:01:54,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] FINISH, SpmStopVDSCommand, log id: 29c94738
>2014-07-26 16:01:54,512 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-26 16:01:54,513 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-26 16:01:54,515 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-26 16:01:54,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] FINISH, ResetIrsVDSCommand, log id: 232547d3
>2014-07-26 16:01:54,545 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-26 16:01:54,559 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] Correlation ID: 6d2260aa, Job ID: 30d80a85-17b3-4966-9eab-0b2c0c124cc7, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-26 16:01:54,568 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-41) [3abdaaa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-26 16:01:54,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-26 16:01:54,641 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-26 16:01:54,667 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [8e871e1] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-26 16:01:54,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [8e871e1] Removing vds [] from the domain in maintenance cache
>2014-07-26 16:01:54,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [8e871e1] Removing host(s) [] from hosts unseen domain report cache
>2014-07-26 16:01:54,676 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-41) [3abdaaa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-26 16:01:57,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5309f983
>2014-07-26 16:01:57,631 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) HostName = rack
>2014-07-26 16:01:57,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-26 16:01:57,649 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-26 16:01:57,657 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-26 16:02:00,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 644182f9
>2014-07-26 16:02:00,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] FINISH, GetHardwareInfoVDSCommand, log id: 644182f9
>2014-07-26 16:02:00,974 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [51c47cce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-26 16:02:01,033 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) [51c47cce] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-26 16:02:01,035 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [2d25bc14] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 16:02:01,042 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-26 16:02:01,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 67765276
>2014-07-26 16:02:01,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 67765276
>2014-07-26 16:02:01,104 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Host rack storage connection was succeeded 
>2014-07-26 16:02:01,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3ed40812
>2014-07-26 16:02:04,511 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-26 16:03:08,236 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, ConnectStoragePoolVDSCommand, log id: 3ed40812
>2014-07-26 16:03:08,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1351da37
>2014-07-26 16:03:08,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1351da37
>2014-07-26 16:03:08,351 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-26 16:03:08,357 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-26 16:03:08,378 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-26 16:03:14,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-26 16:03:14,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-26 16:03:14,864 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39164ee7
>2014-07-26 16:03:14,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling started: taskId = 9cbbbfc5-51f2-47e9-890c-ea207dca555d
>2014-07-26 16:03:17,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended: taskId = 9cbbbfc5-51f2-47e9-890c-ea207dca555d task status = finished
>2014-07-26 16:03:18,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended, spm status: SPM
>2014-07-26 16:03:18,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=9cbbbfc5-51f2-47e9-890c-ea207dca555d), log id: 7a736be4
>2014-07-26 16:03:18,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, HSMClearTaskVDSCommand, log id: 7a736be4
>2014-07-26 16:03:18,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@289da6d6, log id: 39164ee7
>2014-07-26 16:03:18,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-26 16:03:18,078 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-26 16:03:18,080 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-26 16:03:18,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2580f4f8
>2014-07-26 16:03:18,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-26 16:03:18,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 275457d3
>2014-07-26 16:03:18,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 275457d3
>2014-07-26 16:03:18,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2580f4f8
>2014-07-26 16:03:18,171 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) Discovered no tasks on Storage Pool Default
>2014-07-26 16:23:54,492 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-07-26 16:23:59,752 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf.
>2014-07-26 17:23:54,502 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized
>2014-07-26 17:23:59,784 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf.
>2014-07-26 18:23:54,507 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized
>2014-07-26 18:23:59,801 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
>2014-07-26 19:23:54,516 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized
>2014-07-26 19:23:59,818 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf.
>2014-07-26 20:23:54,522 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized
>2014-07-26 20:23:59,833 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf.
>2014-07-26 21:23:54,528 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized
>2014-07-26 21:23:59,864 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf.
>2014-07-26 22:23:54,535 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized
>2014-07-26 22:23:59,876 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf.
>2014-07-26 23:23:54,544 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-07-26 23:23:59,893 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-07-27 00:23:54,549 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized
>2014-07-27 00:23:59,906 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf.
>2014-07-27 01:19:29,341 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 01:19:59,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-27 01:19:59,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-27 01:23:54,555 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-07-27 01:23:59,922 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf.
>2014-07-27 01:24:31,796 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 01:26:05,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 01:26:05,685 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 01:26:05,689 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [341aec49] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-27 01:26:05,706 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [341aec49] Correlation ID: 341aec49, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-27 01:28:05,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 01:28:05,731 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [341aec49] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 01:28:05,733 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-27 01:28:05,737 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 01:28:05,808 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-27 01:28:05,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1e8bf405
>2014-07-27 01:28:05,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 49f1e432
>2014-07-27 01:28:05,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-27 01:30:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering 1 storage domains
>2014-07-27 01:30:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-27 01:30:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-27 01:30:00,015 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/27/14 1:30 AM
>2014-07-27 01:30:00,026 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-27 01:30:00,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 179e9b21
>2014-07-27 01:30:05,699 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-27 01:30:05,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] FINISH, SpmStopVDSCommand, log id: 49f1e432
>2014-07-27 01:30:05,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-27 01:30:05,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 01:30:05,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 01:30:05,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 01:30:05,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] FINISH, ConnectStorageServerVDSCommand, log id: 179e9b21
>2014-07-27 01:30:05,716 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-27 01:30:05,720 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-27 01:30:05,722 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] FINISH, ResetIrsVDSCommand, log id: 1e8bf405
>2014-07-27 01:30:05,721 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 01:30:05,726 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-27 01:30:05,741 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] Correlation ID: 7fa5cf57, Job ID: 2dc1eb12-7c29-446e-95d6-40b9981246dc, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-27 01:30:05,734 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-27 01:30:05,758 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/27/14 1:30 AM
>2014-07-27 01:30:05,763 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-27 01:30:05,764 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [341aec49] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 01:30:05,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-27 01:30:05,826 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-27 01:30:05,849 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [68bbec00] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-27 01:30:05,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [68bbec00] Removing vds [] from the domain in maintenance cache
>2014-07-27 01:30:05,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [68bbec00] Removing host(s) [] from hosts unseen domain report cache
>2014-07-27 01:30:05,858 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [341aec49] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 01:30:08,850 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b06f5c3
>2014-07-27 01:30:08,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) HostName = rack
>2014-07-27 01:30:08,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-27 01:30:08,869 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-27 01:30:08,876 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-27 01:30:12,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5adc3365
>2014-07-27 01:30:12,158 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 5adc3365
>2014-07-27 01:30:12,195 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [3af0095b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 01:30:12,204 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) [3af0095b] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-27 01:30:12,206 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [16ac6dbe] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 01:30:12,214 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 01:30:12,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 754cd96
>2014-07-27 01:30:12,279 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 754cd96
>2014-07-27 01:30:12,280 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Host rack storage connection was succeeded 
>2014-07-27 01:30:12,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 27b06332
>2014-07-27 01:30:15,714 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-27 01:31:12,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 27b06332
>2014-07-27 01:31:12,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 34c8d082
>2014-07-27 01:31:12,814 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 34c8d082
>2014-07-27 01:31:12,822 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-27 01:31:12,828 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-27 01:31:12,843 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 01:31:15,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-27 01:31:15,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-27 01:31:15,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fdd429a
>2014-07-27 01:31:15,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling started: taskId = c940b182-f451-44f2-8e44-2741b374ae40
>2014-07-27 01:31:16,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended: taskId = c940b182-f451-44f2-8e44-2741b374ae40 task status = finished
>2014-07-27 01:31:16,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended, spm status: SPM
>2014-07-27 01:31:16,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c940b182-f451-44f2-8e44-2741b374ae40), log id: 445c5bd6
>2014-07-27 01:31:16,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, HSMClearTaskVDSCommand, log id: 445c5bd6
>2014-07-27 01:31:16,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@151da3d2, log id: 4fdd429a
>2014-07-27 01:31:17,003 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-27 01:31:17,009 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-27 01:31:17,020 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10d611
>2014-07-27 01:31:17,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-27 01:31:17,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2f95a703
>2014-07-27 01:31:17,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f95a703
>2014-07-27 01:31:17,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 10d611
>2014-07-27 01:31:17,053 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) Discovered no tasks on Storage Pool Default
>2014-07-27 02:23:54,562 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized
>2014-07-27 02:23:59,938 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-07-27 03:23:54,572 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-07-27 03:23:59,971 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf.
>2014-07-27 03:35:35,000 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-86) Start deleteAgedOutAuditLogs
>2014-07-27 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Start deleteAgedOutCommandEntities
>2014-07-27 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-86) Finished deleteAgedOutAuditLogs
>2014-07-27 03:35:35,008 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Finished deleteAgedOutCommandEntities
>2014-07-27 04:23:54,579 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized
>2014-07-27 04:24:00,003 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf.
>2014-07-27 05:23:54,589 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized
>2014-07-27 05:24:00,019 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf.
>2014-07-27 06:00:30,041 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-27 06:23:54,595 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized
>2014-07-27 06:24:00,050 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf.
>2014-07-27 07:23:54,601 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized
>2014-07-27 07:24:00,066 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf.
>2014-07-27 08:23:54,610 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [375fcc6] Principal admin::internal synchronized
>2014-07-27 08:24:00,080 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf.
>2014-07-27 09:23:54,617 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized
>2014-07-27 09:24:00,110 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf.
>2014-07-27 10:23:54,623 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized
>2014-07-27 10:24:00,123 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-27 10:37:28,171 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 10:39:03,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 10:39:03,270 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 10:39:03,276 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-62) [2e742a17] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-27 10:39:03,292 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [2e742a17] Correlation ID: 2e742a17, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-27 10:41:03,307 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 10:41:03,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2e742a17] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 10:41:03,311 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-27 10:41:03,312 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 10:41:03,358 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-27 10:41:03,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 56499c77
>2014-07-27 10:41:03,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 591fa6b6
>2014-07-27 10:41:03,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-27 10:43:03,281 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-27 10:43:03,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] FINISH, SpmStopVDSCommand, log id: 591fa6b6
>2014-07-27 10:43:03,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 10:43:03,290 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-27 10:43:03,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-27 10:43:03,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 10:43:03,294 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 10:43:03,294 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] FINISH, ResetIrsVDSCommand, log id: 56499c77
>2014-07-27 10:43:03,309 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [606e9025] Correlation ID: 606e9025, Job ID: 4a0911e4-efb5-4e19-a046-77c333224b30, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-27 10:43:03,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-27 10:43:03,319 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [2e742a17] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 10:43:03,341 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-27 10:43:03,344 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-27 10:43:03,402 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-27 10:43:03,429 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-27 10:43:03,436 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] Removing vds [] from the domain in maintenance cache
>2014-07-27 10:43:03,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] Removing host(s) [] from hosts unseen domain report cache
>2014-07-27 10:43:03,438 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [2e742a17] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 10:43:06,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 61f0f4eb
>2014-07-27 10:43:06,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GetHardwareInfoVDSCommand, log id: 61f0f4eb
>2014-07-27 10:43:06,925 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [4f0c66c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 10:43:06,975 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [71c44c7e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 10:43:06,984 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 10:43:06,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6235aea6
>2014-07-27 10:43:07,052 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 6235aea6
>2014-07-27 10:43:07,054 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] Host rack storage connection was succeeded 
>2014-07-27 10:43:07,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 465ca7d8
>2014-07-27 10:43:13,300 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-27 10:44:11,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) FINISH, ConnectStoragePoolVDSCommand, log id: 465ca7d8
>2014-07-27 10:44:11,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a6c2703
>2014-07-27 10:44:11,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a6c2703
>2014-07-27 10:44:11,648 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [5909f95c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-27 10:44:11,654 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [5909f95c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-27 10:44:11,670 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 10:44:13,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-27 10:44:13,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-27 10:44:13,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3905a945
>2014-07-27 10:44:13,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling started: taskId = 0d169a53-0a63-48cd-9d32-55c245254da1
>2014-07-27 10:44:15,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended: taskId = 0d169a53-0a63-48cd-9d32-55c245254da1 task status = finished
>2014-07-27 10:44:15,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended, spm status: SPM
>2014-07-27 10:44:15,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=0d169a53-0a63-48cd-9d32-55c245254da1), log id: 57836cb1
>2014-07-27 10:44:16,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, HSMClearTaskVDSCommand, log id: 57836cb1
>2014-07-27 10:44:16,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63f08866, log id: 3905a945
>2014-07-27 10:44:16,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-27 10:44:16,042 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-27 10:44:16,043 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-27 10:44:16,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5a9a3132
>2014-07-27 10:44:16,080 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-27 10:44:16,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 47cdeb08
>2014-07-27 10:44:16,087 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47cdeb08
>2014-07-27 10:44:16,088 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a9a3132
>2014-07-27 10:44:16,089 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) Discovered no tasks on Storage Pool Default
>2014-07-27 11:23:54,628 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized
>2014-07-27 11:24:00,138 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
>2014-07-27 12:23:54,636 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized
>2014-07-27 12:24:00,170 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-07-27 13:23:54,642 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized
>2014-07-27 13:24:00,201 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf.
>2014-07-27 14:23:54,648 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized
>2014-07-27 14:24:00,213 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-07-27 14:55:01,130 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 6.1082 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-27 15:23:54,653 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized
>2014-07-27 15:24:00,226 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf.
>2014-07-27 16:23:54,670 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized
>2014-07-27 16:24:00,241 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-27 16:31:05,625 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-16) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-27 17:23:54,678 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized
>2014-07-27 17:24:00,272 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
>2014-07-27 18:23:54,683 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized
>2014-07-27 18:24:00,286 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-07-27 18:30:00,449 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 6.15416 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-27 19:05:02,358 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 19:05:47,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-27 19:05:47,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-27 19:23:54,689 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized
>2014-07-27 19:24:00,304 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf.
>2014-07-27 20:07:02,563 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 20:08:30,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 20:08:30,119 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 20:08:30,122 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-27 20:08:30,134 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [52dee25c] Correlation ID: 52dee25c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-27 20:10:30,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 20:10:30,165 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [52dee25c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 20:10:30,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-27 20:10:30,168 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-27 20:10:30,200 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-27 20:10:30,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 27e1459d
>2014-07-27 20:10:30,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 768443a3
>2014-07-27 20:10:30,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-27 20:12:30,134 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-27 20:12:30,138 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] FINISH, SpmStopVDSCommand, log id: 768443a3
>2014-07-27 20:12:30,149 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-27 20:12:30,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-27 20:12:30,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] FINISH, ResetIrsVDSCommand, log id: 27e1459d
>2014-07-27 20:12:30,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-27 20:12:30,162 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-27 20:12:30,163 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-27 20:12:30,167 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [40b4906d] Correlation ID: 40b4906d, Job ID: b3f62e68-097e-4c0b-b35a-0124dfe278b4, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-27 20:12:30,202 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [52dee25c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 20:12:30,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-27 20:12:30,206 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-27 20:12:30,274 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-27 20:12:30,303 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [57c73f48] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-27 20:12:30,312 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [57c73f48] Removing vds [] from the domain in maintenance cache
>2014-07-27 20:12:30,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [57c73f48] Removing host(s) [] from hosts unseen domain report cache
>2014-07-27 20:12:30,314 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [52dee25c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-27 20:12:33,295 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@46ea165b
>2014-07-27 20:12:33,297 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) HostName = rack
>2014-07-27 20:12:33,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-27 20:12:33,324 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-27 20:12:33,331 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-27 20:12:36,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 78c1c62b
>2014-07-27 20:12:36,626 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GetHardwareInfoVDSCommand, log id: 78c1c62b
>2014-07-27 20:12:36,670 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [1df66af2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 20:12:36,679 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) [1df66af2] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-27 20:12:36,682 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-100) [685f7cc1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 20:12:36,693 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-27 20:12:36,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d4ff2cf
>2014-07-27 20:12:36,762 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: d4ff2cf
>2014-07-27 20:12:36,763 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Host rack storage connection was succeeded 
>2014-07-27 20:12:36,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 227014c
>2014-07-27 20:12:40,161 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-27 20:13:37,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, ConnectStoragePoolVDSCommand, log id: 227014c
>2014-07-27 20:13:37,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 579b07b1
>2014-07-27 20:13:37,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 579b07b1
>2014-07-27 20:13:37,330 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-27 20:13:37,337 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-27 20:13:37,356 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-27 20:13:40,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-27 20:13:40,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-27 20:13:40,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 720c77db
>2014-07-27 20:13:40,331 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = dee365f4-3ae8-417e-93b4-914603a97a8c
>2014-07-27 20:13:41,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = dee365f4-3ae8-417e-93b4-914603a97a8c task status = finished
>2014-07-27 20:13:41,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM
>2014-07-27 20:13:41,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=dee365f4-3ae8-417e-93b4-914603a97a8c), log id: 5dedde83
>2014-07-27 20:13:41,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 5dedde83
>2014-07-27 20:13:41,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b28499b, log id: 720c77db
>2014-07-27 20:13:41,446 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-27 20:13:41,451 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-27 20:13:41,461 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1623a10b
>2014-07-27 20:13:41,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-27 20:13:41,491 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1666e8a4
>2014-07-27 20:13:41,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1666e8a4
>2014-07-27 20:13:41,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1623a10b
>2014-07-27 20:13:41,498 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) Discovered no tasks on Storage Pool Default
>2014-07-27 20:23:54,696 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized
>2014-07-27 20:24:00,320 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf.
>2014-07-27 20:45:41,027 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-27 20:45:56,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-27 20:45:56,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-27 21:23:54,703 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized
>2014-07-27 21:24:00,336 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf.
>2014-07-27 21:28:10,510 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-22) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-27 22:23:54,712 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized
>2014-07-27 22:24:00,367 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf.
>2014-07-27 23:23:54,718 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized
>2014-07-27 23:24:00,380 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-07-28 00:23:54,734 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-07-28 00:24:00,393 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf.
>2014-07-28 01:23:54,739 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2734597c] Principal admin::internal synchronized
>2014-07-28 01:24:00,409 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf.
>2014-07-28 02:23:54,746 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized
>2014-07-28 02:24:00,421 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf.
>2014-07-28 03:23:54,750 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized
>2014-07-28 03:24:00,434 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
>2014-07-28 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Start deleteAgedOutAuditLogs
>2014-07-28 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Start deleteAgedOutCommandEntities
>2014-07-28 03:35:35,003 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Finished deleteAgedOutAuditLogs
>2014-07-28 03:35:35,009 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Finished deleteAgedOutCommandEntities
>2014-07-28 04:23:54,758 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized
>2014-07-28 04:24:00,448 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf.
>2014-07-28 05:23:54,764 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized
>2014-07-28 05:24:00,495 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-07-28 05:24:53,022 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-28 05:26:22,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 05:26:22,072 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 05:26:22,075 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [75d6be08] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-28 05:26:22,108 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [75d6be08] Correlation ID: 75d6be08, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-28 05:28:22,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 05:28:22,120 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [75d6be08] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-28 05:28:22,121 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-28 05:28:22,122 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 05:28:22,163 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-28 05:28:22,180 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4baa9eba
>2014-07-28 05:28:22,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 69a518e5
>2014-07-28 05:28:22,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-28 05:30:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) Autorecovering 1 storage domains
>2014-07-28 05:30:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-28 05:30:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-28 05:30:00,013 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/28/14 5:30 AM
>2014-07-28 05:30:00,024 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-28 05:30:00,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 497b3dd
>2014-07-28 05:30:22,097 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-28 05:30:22,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] FINISH, SpmStopVDSCommand, log id: 69a518e5
>2014-07-28 05:30:22,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-28 05:30:22,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-28 05:30:22,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-54) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-28 05:30:22,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [60dd31b0] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-28 05:30:22,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] FINISH, ConnectStorageServerVDSCommand, log id: 497b3dd
>2014-07-28 05:30:22,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] FINISH, ResetIrsVDSCommand, log id: 4baa9eba
>2014-07-28 05:30:22,116 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-28 05:30:22,118 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-28 05:30:22,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-28 05:30:22,134 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [44b05f41] Correlation ID: 44b05f41, Job ID: a958ccca-6f65-4e38-b975-60c971a1630c, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-28 05:30:22,134 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-28 05:30:22,156 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-28 05:30:22,158 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/28/14 5:30 AM
>2014-07-28 05:30:22,160 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-3) [75d6be08] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-28 05:30:22,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-28 05:30:22,249 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-28 05:30:22,276 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [1892618e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-28 05:30:22,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [1892618e] Removing vds [] from the domain in maintenance cache
>2014-07-28 05:30:22,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [1892618e] Removing host(s) [] from hosts unseen domain report cache
>2014-07-28 05:30:22,286 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-3) [75d6be08] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-28 05:30:25,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27f19f5b
>2014-07-28 05:30:25,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) HostName = rack
>2014-07-28 05:30:25,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-28 05:30:25,256 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-28 05:30:25,263 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-28 05:30:28,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@10ac4b2c
>2014-07-28 05:30:28,278 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) HostName = rack
>2014-07-28 05:30:28,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-28 05:30:31,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 10939650
>2014-07-28 05:30:31,560 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 10939650
>2014-07-28 05:30:31,585 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [7f2483a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-28 05:30:31,636 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) [7f2483a0] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-28 05:30:31,638 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [4beb5dac] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 05:30:31,646 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 05:30:31,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cd4bf1e
>2014-07-28 05:30:31,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 6cd4bf1e
>2014-07-28 05:30:31,716 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Host rack storage connection was succeeded 
>2014-07-28 05:30:31,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bc4b16
>2014-07-28 05:30:32,122 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-28 05:31:38,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 2bc4b16
>2014-07-28 05:31:38,731 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 767366e7
>2014-07-28 05:31:38,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] FINISH, SetMOMPolicyParametersVDSCommand, log id: 767366e7
>2014-07-28 05:31:38,831 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-28 05:31:38,837 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-28 05:31:38,856 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-28 05:31:42,570 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-28 05:31:42,578 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-28 05:31:42,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3474374
>2014-07-28 05:31:42,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling started: taskId = 16f6da0a-6893-4a05-acb1-d3a4342979ac
>2014-07-28 05:31:46,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling ended: taskId = 16f6da0a-6893-4a05-acb1-d3a4342979ac task status = finished
>2014-07-28 05:31:46,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling ended, spm status: SPM
>2014-07-28 05:31:46,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-45) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=16f6da0a-6893-4a05-acb1-d3a4342979ac), log id: 2a5c481b
>2014-07-28 05:31:46,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, HSMClearTaskVDSCommand, log id: 2a5c481b
>2014-07-28 05:31:46,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a246513, log id: 3474374
>2014-07-28 05:31:46,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-28 05:31:46,807 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-28 05:31:46,809 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-28 05:31:46,820 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65e5fbdf
>2014-07-28 05:31:47,132 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-28 05:31:47,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 957fbd5
>2014-07-28 05:31:47,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 957fbd5
>2014-07-28 05:31:47,142 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 65e5fbdf
>2014-07-28 05:31:47,144 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default
>2014-07-28 06:23:54,772 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized
>2014-07-28 06:24:00,510 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf.
>2014-07-28 07:23:54,780 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized
>2014-07-28 07:24:00,523 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-07-28 08:23:54,786 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized
>2014-07-28 08:24:00,542 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-07-28 09:23:54,792 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized
>2014-07-28 09:24:00,556 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf.
>2014-07-28 10:23:54,797 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized
>2014-07-28 10:24:00,572 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf.
>2014-07-28 11:23:54,803 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized
>2014-07-28 11:24:00,588 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-28 12:23:54,810 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2734597c] Principal admin::internal synchronized
>2014-07-28 12:24:00,624 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
>2014-07-28 13:23:54,817 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) Principal admin::internal synchronized
>2014-07-28 13:24:00,655 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf.
>2014-07-28 14:23:54,822 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized
>2014-07-28 14:24:00,670 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf.
>2014-07-28 14:53:39,769 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-28 14:55:14,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 14:55:14,717 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 14:55:14,722 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-28 14:55:14,735 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Correlation ID: 2f0280a3, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-28 14:57:14,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 14:57:14,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-28 14:57:14,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-28 14:57:14,764 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-28 14:57:14,795 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-28 14:57:14,813 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5ca7f0a4
>2014-07-28 14:57:14,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 948523b
>2014-07-28 14:57:14,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-28 14:59:14,729 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-28 14:59:14,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] FINISH, SpmStopVDSCommand, log id: 948523b
>2014-07-28 14:59:14,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-28 14:59:14,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-28 14:59:14,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] FINISH, ResetIrsVDSCommand, log id: 5ca7f0a4
>2014-07-28 14:59:14,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-28 14:59:14,744 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-28 14:59:14,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-28 14:59:14,752 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-28 14:59:14,765 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [59aadc32] Correlation ID: 59aadc32, Job ID: ef052d9e-7451-4f4b-b3d1-b767dad6b2e5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-28 14:59:14,793 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-28 14:59:14,797 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [2f0280a3] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-28 14:59:14,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-28 14:59:14,865 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-28 14:59:14,894 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [62148fba] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-28 14:59:14,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [62148fba] Removing vds [] from the domain in maintenance cache
>2014-07-28 14:59:14,903 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [62148fba] Removing host(s) [] from hosts unseen domain report cache
>2014-07-28 14:59:14,904 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [2f0280a3] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-28 14:59:18,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 510a2203
>2014-07-28 14:59:18,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GetHardwareInfoVDSCommand, log id: 510a2203
>2014-07-28 14:59:18,129 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [3af88bf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-28 14:59:18,178 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [25ea9bee] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 14:59:18,186 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-28 14:59:18,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 674689c4
>2014-07-28 14:59:18,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 674689c4
>2014-07-28 14:59:18,248 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Host rack storage connection was succeeded 
>2014-07-28 14:59:18,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 71f70b65
>2014-07-28 14:59:24,750 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-28 15:00:18,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 71f70b65
>2014-07-28 15:00:18,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 10685ea8
>2014-07-28 15:00:18,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 10685ea8
>2014-07-28 15:00:18,731 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-28 15:00:18,738 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-28 15:00:18,754 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-28 15:00:24,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-28 15:00:24,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-28 15:00:24,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34a8334f
>2014-07-28 15:00:24,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling started: taskId = a794c812-083e-47c3-a1c0-972895bb6e6b
>2014-07-28 15:00:25,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended: taskId = a794c812-083e-47c3-a1c0-972895bb6e6b task status = finished
>2014-07-28 15:00:25,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended, spm status: SPM
>2014-07-28 15:00:25,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=a794c812-083e-47c3-a1c0-972895bb6e6b), log id: 403883bd
>2014-07-28 15:00:26,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 403883bd
>2014-07-28 15:00:26,019 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4387eab8, log id: 34a8334f
>2014-07-28 15:00:26,023 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-28 15:00:26,029 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-28 15:00:26,045 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 36d29e43
>2014-07-28 15:00:26,069 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-28 15:00:26,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1bf40fba
>2014-07-28 15:00:26,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1bf40fba
>2014-07-28 15:00:26,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36d29e43
>2014-07-28 15:00:26,078 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-35) Discovered no tasks on Storage Pool Default
>2014-07-28 15:23:54,827 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized
>2014-07-28 15:24:00,700 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf.
>2014-07-28 16:23:54,831 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized
>2014-07-28 16:24:00,715 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
>2014-07-28 16:30:59,760 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-28 16:31:14,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-28 16:31:14,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-28 17:23:54,836 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized
>2014-07-28 17:24:00,729 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
>2014-07-28 18:23:54,844 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Principal admin::internal synchronized
>2014-07-28 18:24:00,743 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf.
>2014-07-28 19:23:54,851 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized
>2014-07-28 19:24:00,780 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf.
>2014-07-28 20:23:54,857 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-07-28 20:24:00,794 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf.
>2014-07-28 21:00:31,448 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-28 21:23:54,863 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-07-28 21:24:00,811 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [75d6be08] Attempting to update VMs/Templates Ovf.
>2014-07-28 22:23:54,872 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized
>2014-07-28 22:24:00,823 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf.
>2014-07-28 23:23:54,878 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-07-28 23:24:00,852 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-07-29 00:23:18,509 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 00:23:54,885 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized
>2014-07-29 00:24:00,899 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf.
>2014-07-29 00:24:50,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 00:24:50,903 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 00:24:50,907 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-29 00:24:50,920 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Correlation ID: 738ebb7b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-29 00:26:50,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 00:26:50,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [738ebb7b] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 00:26:50,952 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-29 00:26:50,953 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 00:26:50,988 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-29 00:26:51,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2b83c9f
>2014-07-29 00:26:51,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 78bbba55
>2014-07-29 00:26:51,015 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-29 00:28:50,921 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-29 00:28:50,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-29 00:28:50,927 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 00:28:50,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-24) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 00:28:50,931 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 00:28:50,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] FINISH, SpmStopVDSCommand, log id: 78bbba55
>2014-07-29 00:28:50,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 00:28:50,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] FINISH, ResetIrsVDSCommand, log id: 2b83c9f
>2014-07-29 00:28:50,967 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 00:28:50,976 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [7b0933af] Correlation ID: 7b0933af, Job ID: bad0c9c9-9544-4d30-84c9-8154504ac4e8, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-29 00:28:50,982 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [738ebb7b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 00:28:50,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-29 00:28:51,047 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-29 00:28:51,067 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [380ca791] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-29 00:28:51,074 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [380ca791] Removing vds [] from the domain in maintenance cache
>2014-07-29 00:28:51,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [380ca791] Removing host(s) [] from hosts unseen domain report cache
>2014-07-29 00:28:51,076 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [738ebb7b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 00:28:54,232 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 614f08b0
>2014-07-29 00:28:54,286 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 614f08b0
>2014-07-29 00:28:54,317 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [28f420dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 00:28:54,334 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [7f5d96ca] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 00:28:54,342 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 00:28:54,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5686beb8
>2014-07-29 00:28:54,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5686beb8
>2014-07-29 00:28:54,408 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Host rack storage connection was succeeded 
>2014-07-29 00:28:54,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6071a758
>2014-07-29 00:29:00,931 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-9) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-29 00:29:54,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, ConnectStoragePoolVDSCommand, log id: 6071a758
>2014-07-29 00:29:54,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ef86907
>2014-07-29 00:29:54,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1ef86907
>2014-07-29 00:29:54,924 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-29 00:29:54,930 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 00:29:54,950 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 00:30:01,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-29 00:30:01,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-29 00:30:01,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7a87537
>2014-07-29 00:30:01,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling started: taskId = 8fcd82c5-f85c-4929-9e85-494fda6c75ed
>2014-07-29 00:30:05,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended: taskId = 8fcd82c5-f85c-4929-9e85-494fda6c75ed task status = finished
>2014-07-29 00:30:05,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended, spm status: SPM
>2014-07-29 00:30:05,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=8fcd82c5-f85c-4929-9e85-494fda6c75ed), log id: be7e3f5
>2014-07-29 00:30:05,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: be7e3f5
>2014-07-29 00:30:05,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@669cc87, log id: 7a87537
>2014-07-29 00:30:05,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-29 00:30:05,595 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-29 00:30:05,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7159591f
>2014-07-29 00:30:05,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-29 00:30:05,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d198902
>2014-07-29 00:30:05,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d198902
>2014-07-29 00:30:05,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7159591f
>2014-07-29 00:30:05,644 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-25) Discovered no tasks on Storage Pool Default
>2014-07-29 01:23:54,893 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized
>2014-07-29 01:24:00,917 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
>2014-07-29 02:23:54,900 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized
>2014-07-29 02:24:00,930 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf.
>2014-07-29 03:23:54,907 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) Principal admin::internal synchronized
>2014-07-29 03:24:00,945 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf.
>2014-07-29 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-97) Start deleteAgedOutCommandEntities
>2014-07-29 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Start deleteAgedOutAuditLogs
>2014-07-29 03:35:35,005 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Finished deleteAgedOutAuditLogs
>2014-07-29 03:35:35,007 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-97) Finished deleteAgedOutCommandEntities
>2014-07-29 04:15:31,723 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-5) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-29 04:23:54,912 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized
>2014-07-29 04:24:00,960 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf.
>2014-07-29 05:23:54,919 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized
>2014-07-29 05:24:00,973 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-29 06:23:54,924 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-07-29 06:24:00,988 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-07-29 07:23:54,929 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized
>2014-07-29 07:24:01,001 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf.
>2014-07-29 07:24:15,032 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.7491 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-29 08:23:54,935 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized
>2014-07-29 08:24:01,017 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf.
>2014-07-29 09:23:54,940 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized
>2014-07-29 09:24:01,032 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf.
>2014-07-29 09:42:37,236 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 09:44:02,211 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 09:44:02,215 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 09:44:02,219 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-29 09:44:02,233 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Correlation ID: 5d8b5376, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-29 09:46:02,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 09:46:02,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 09:46:02,261 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-29 09:46:02,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 09:46:02,304 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-29 09:46:02,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6d2622c
>2014-07-29 09:46:02,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f5b25d8
>2014-07-29 09:46:02,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-29 09:48:02,228 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-29 09:48:02,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] FINISH, SpmStopVDSCommand, log id: 5f5b25d8
>2014-07-29 09:48:02,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-29 09:48:02,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 09:48:02,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 09:48:02,237 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 09:48:02,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 09:48:02,248 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] FINISH, ResetIrsVDSCommand, log id: 6d2622c
>2014-07-29 09:48:02,271 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 09:48:02,272 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] Correlation ID: 1d3e693b, Job ID: aefe862a-a37f-4347-ba5e-e324aebf0661, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-29 09:48:02,279 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [5d8b5376] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 09:48:02,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-29 09:48:02,342 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-29 09:48:02,361 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-18) [7dffa625] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-29 09:48:02,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [7dffa625] Removing vds [] from the domain in maintenance cache
>2014-07-29 09:48:02,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [7dffa625] Removing host(s) [] from hosts unseen domain report cache
>2014-07-29 09:48:02,370 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [5d8b5376] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 09:48:05,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3b7798a4
>2014-07-29 09:48:05,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] FINISH, GetHardwareInfoVDSCommand, log id: 3b7798a4
>2014-07-29 09:48:05,637 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [51f62f58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 09:48:05,649 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [46b071d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 09:48:05,657 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 09:48:05,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50722f49
>2014-07-29 09:48:05,730 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 50722f49
>2014-07-29 09:48:05,731 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] Host rack storage connection was succeeded 
>2014-07-29 09:48:05,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7ae58a9c
>2014-07-29 09:48:12,237 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-29 09:49:14,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 7ae58a9c
>2014-07-29 09:49:14,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ba9f8de
>2014-07-29 09:49:14,210 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1ba9f8de
>2014-07-29 09:49:14,218 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [c3cc627] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-29 09:49:14,225 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [c3cc627] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 09:49:14,242 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 09:49:22,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-29 09:49:22,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-29 09:49:22,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 17059a88
>2014-07-29 09:49:22,412 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling started: taskId = 2621acf2-1ca0-453a-b925-4b9ff956e6e9
>2014-07-29 09:49:23,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended: taskId = 2621acf2-1ca0-453a-b925-4b9ff956e6e9 task status = finished
>2014-07-29 09:49:23,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended, spm status: SPM
>2014-07-29 09:49:23,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=2621acf2-1ca0-453a-b925-4b9ff956e6e9), log id: 1800b4f4
>2014-07-29 09:49:23,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, HSMClearTaskVDSCommand, log id: 1800b4f4
>2014-07-29 09:49:23,523 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57c761b7, log id: 17059a88
>2014-07-29 09:49:23,527 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-29 09:49:23,534 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-29 09:49:23,536 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-24) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-29 09:49:23,547 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 670d4aca
>2014-07-29 09:49:23,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-29 09:49:23,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 532d9fb4
>2014-07-29 09:49:23,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 532d9fb4
>2014-07-29 09:49:23,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 670d4aca
>2014-07-29 09:49:23,578 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-24) Discovered no tasks on Storage Pool Default
>2014-07-29 10:23:54,947 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized
>2014-07-29 10:24:01,047 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
>2014-07-29 10:28:11,818 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
>2014-07-29 10:28:11,824 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:28:12,477 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-29 10:49:12,454 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-07-29 10:49:12,459 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:49:13,349 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [3694f201] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:49:13,356 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [6a24efe7] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:49:13,362 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [6a24efe7] Correlation ID: 6a24efe7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:49:13,769 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-07-29 10:49:13,783 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [3ebcd9a] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:49:13,787 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [579bdb1b] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:49:13,792 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [579bdb1b] Correlation ID: 579bdb1b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:49:26,513 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-29 10:49:26,517 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:49:26,547 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [1dbb305f] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:49:26,552 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [180692ea] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:49:26,563 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [180692ea] Correlation ID: 180692ea, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:49:46,382 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-07-29 10:49:46,386 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:49:48,310 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [5bcc8676] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:49:48,312 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4fc67a4c] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:49:48,316 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4fc67a4c] Correlation ID: 4fc67a4c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:50:44,485 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-29 10:50:44,489 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:50:45,275 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [3a9f0743] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:50:45,278 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [65464a80] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:50:45,282 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [65464a80] Correlation ID: 65464a80, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:50:45,625 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-07-29 10:50:45,665 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [44d6b145] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:50:45,667 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [48b3bc09] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:50:45,670 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [48b3bc09] Correlation ID: 48b3bc09, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:52:27,795 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-29 10:52:27,801 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:52:27,808 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [38f821c0] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:52:27,813 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [5f8314d9] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:52:27,818 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5f8314d9] Correlation ID: 5f8314d9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:52:36,559 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-07-29 10:52:36,563 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:52:37,503 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [30b339af] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:52:37,505 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [3ded41b5] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:52:37,509 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3ded41b5] Correlation ID: 3ded41b5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:53:10,058 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-07-29 10:53:10,061 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:54:40,251 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-29 10:54:40,260 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:54:40,267 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [31145d7b] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:54:40,270 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [6cbf2fba] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:54:40,274 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6cbf2fba] Correlation ID: 6cbf2fba, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 10:55:40,177 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-07-29 10:55:40,181 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 10:55:40,187 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [6cb66736] Running command: LogoutBySessionCommand internal: false.
>2014-07-29 10:55:40,196 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [34524ed8] Running command: LogoutUserCommand internal: false.
>2014-07-29 10:55:40,203 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [34524ed8] Correlation ID: 34524ed8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-29 11:23:54,956 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized
>2014-07-29 11:24:01,062 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf.
>2014-07-29 12:23:54,962 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized
>2014-07-29 12:24:01,076 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) [4dca0641] Attempting to update VMs/Templates Ovf.
>2014-07-29 12:58:21,049 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 12:58:36,139 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-29 12:58:36,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-29 12:59:21,621 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.60819 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-29 13:23:54,966 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized
>2014-07-29 13:24:01,088 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
>2014-07-29 14:00:21,061 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 14:00:36,191 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-29 14:00:36,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-29 14:01:21,593 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 7.1651 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-29 14:23:54,973 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized
>2014-07-29 14:24:01,102 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf.
>2014-07-29 14:25:48,963 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 14:26:04,088 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-29 14:26:04,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-29 15:23:54,980 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized
>2014-07-29 15:24:01,117 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf.
>2014-07-29 15:44:11,493 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.1973 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-29 16:05:06,478 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 16:05:21,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-29 16:05:21,603 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-29 16:23:54,986 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized
>2014-07-29 16:24:01,148 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf.
>2014-07-29 16:47:17,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:17,749 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 16:47:17,756 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-29 16:47:17,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:17,775 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 16:47:17,784 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: 7e0971f4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Network error during communication with the Host.).
>2014-07-29 16:47:17,778 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 16:47:17,809 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused
>2014-07-29 16:47:17,842 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 16:47:20,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5c539198
>2014-07-29 16:47:20,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] HostName = rack
>2014-07-29 16:47:20,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-29 16:47:20,960 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [72971053] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-29 16:47:20,967 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [72971053] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-29 16:47:24,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 64603823
>2014-07-29 16:47:24,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GetHardwareInfoVDSCommand, log id: 64603823
>2014-07-29 16:47:24,346 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [e91c5b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 16:47:24,355 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) [e91c5b2] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-29 16:47:24,357 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [1134a0fc] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 16:47:24,369 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 16:47:24,380 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5f7ebc9b
>2014-07-29 16:47:24,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5f7ebc9b
>2014-07-29 16:47:24,437 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] Host rack storage connection was succeeded 
>2014-07-29 16:47:24,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2600cee
>2014-07-29 16:47:27,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
>
>2014-07-29 16:47:27,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] HostName = rack
>2014-07-29 16:47:27,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
>2014-07-29 16:47:39,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-29 16:47:39,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 2600cee
>2014-07-29 16:47:39,316 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-21) Could not connect host rack to pool Default, as the master domain is in inactive/unknown status - not failing the operation
>2014-07-29 16:47:39,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:39,322 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-21) Could not get Host statistics for Host rack, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:]
>	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) [bll.jar:]
>	at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:627) [bll.jar:]
>	at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:]
>	at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:]
>	at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:]
>	at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:]
>	at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 16:47:39,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack reports about one of the Active Storage Domains as Problematic.
>2014-07-29 16:47:39,358 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 16:47:39,362 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3a0d438e
>2014-07-29 16:47:39,364 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] VDS rack is spm and moved from up calling resetIrs.
>2014-07-29 16:47:39,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 2f5ddc2e
>2014-07-29 16:47:39,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 59f7ce8a
>2014-07-29 16:47:39,373 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:39,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-29 16:47:39,382 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-29 16:47:39,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, SpmStopVDSCommand, log id: 59f7ce8a
>2014-07-29 16:47:39,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, ResetIrsVDSCommand, log id: 2f5ddc2e
>2014-07-29 16:47:39,391 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, SetVdsStatusVDSCommand, log id: 3a0d438e
>2014-07-29 16:47:39,415 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: 654b0a57, Job ID: 769ebd00-e82d-4a7c-8271-5b8eb1c0e432, Call Stack: null, Custom Event ID: -1, Message: Host rack cannot access the Storage Domain(s) Non interactive user attached to the Data Center Default. Setting Host state to Non-Operational.
>2014-07-29 16:47:39,428 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 16:47:39,435 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: 1134a0fc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to Storage Pool Default
>2014-07-29 16:47:39,452 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 16:47:39,454 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [4760e42] Host 17f7fe56-5f32-40ef-b098-611b84484e0b : rack is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.
>2014-07-29 16:47:39,490 INFO  [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-29 16:47:39,554 WARN  [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] CanDoAction of action InternalMigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_NO_HOSTS
>2014-07-29 16:47:39,555 INFO  [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-29 16:47:39,563 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] ResourceManager::vdsMaintenance - Failed migrating desktop testtesttest
>2014-07-29 16:47:42,465 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:42,468 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 16:47:42,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:42,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 16:47:42,501 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 16:47:45,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 16:47:45,512 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 16:47:45,513 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsCapabilities(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 16:47:45,543 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 16:47:48,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e309e7f
>2014-07-29 16:47:48,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = rack
>2014-07-29 16:47:48,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-29 16:47:48,665 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-29 16:47:52,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3c8a726d
>2014-07-29 16:47:52,147 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, GetHardwareInfoVDSCommand, log id: 3c8a726d
>2014-07-29 16:47:52,171 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [2160db07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 16:47:52,221 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [2160db07] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-29 16:47:52,224 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [78371c83] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 16:47:52,231 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 16:47:52,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38b36d36
>2014-07-29 16:47:52,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 38b36d36
>2014-07-29 16:47:52,292 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Host rack storage connection was succeeded 
>2014-07-29 16:47:52,299 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7a80b6fd
>2014-07-29 16:48:52,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 7a80b6fd
>2014-07-29 16:48:52,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5756c566
>2014-07-29 16:48:53,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5756c566
>2014-07-29 16:48:53,025 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-29 16:48:53,031 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 16:48:53,050 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 16:48:58,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-29 16:48:58,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-29 16:48:58,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54a9cff5
>2014-07-29 16:48:58,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling started: taskId = 293fb8a0-656c-4123-aacc-626905de7937
>2014-07-29 16:48:59,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling ended: taskId = 293fb8a0-656c-4123-aacc-626905de7937 task status = finished
>2014-07-29 16:48:59,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling ended, spm status: SPM
>2014-07-29 16:48:59,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=293fb8a0-656c-4123-aacc-626905de7937), log id: 2fa7097b
>2014-07-29 16:48:59,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, HSMClearTaskVDSCommand, log id: 2fa7097b
>2014-07-29 16:48:59,276 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dab6934, log id: 54a9cff5
>2014-07-29 16:48:59,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-29 16:48:59,286 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-29 16:48:59,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f8caf47
>2014-07-29 16:48:59,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-29 16:48:59,323 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70aad3f8
>2014-07-29 16:48:59,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 70aad3f8
>2014-07-29 16:48:59,330 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f8caf47
>2014-07-29 16:48:59,331 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-50) Discovered no tasks on Storage Pool Default
>2014-07-29 17:03:10,268 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-29 17:03:10,334 WARN  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,CANNOT_MAINTENANCE_VDS_RUN_VMS_NO_OTHER_RUNNING_VDS,$ClustersList Default,$HostsList rack
>2014-07-29 17:03:10,336 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-29 17:03:24,889 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [95994fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-29 17:03:24,986 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] Running command: RunVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-29 17:03:24,993 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] START, ResumeVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e), log id: 7be2f515
>2014-07-29 17:03:24,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] START, ResumeBrokerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e), log id: 1c7bdac5
>2014-07-29 17:03:25,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] FINISH, ResumeBrokerVDSCommand, log id: 1c7bdac5
>2014-07-29 17:03:25,047 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] FINISH, ResumeVDSCommand, return: PoweringUp, log id: 7be2f515
>2014-07-29 17:03:25,048 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-07-29 17:03:25,063 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [95994fa] Correlation ID: 95994fa, Job ID: ab7eff79-8541-41d9-99c3-000b056e06c8, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was resumed by admin (Host: rack).
>2014-07-29 17:03:27,533 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Paused --> Up
>2014-07-29 17:03:27,547 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: 95994fa, Job ID: ab7eff79-8541-41d9-99c3-000b056e06c8, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack
>2014-07-29 17:05:40,708 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Running command: ShutdownVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-07-29 17:05:40,712 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Entered (VM testtesttest).
>2014-07-29 17:05:40,713 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Sending shutdown command for VM testtesttest.
>2014-07-29 17:05:40,716 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=30, gracefully=true, reason=), log id: 5e9a73e1
>2014-07-29 17:05:40,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=30, gracefully=true, reason=), log id: 9d52f7b
>2014-07-29 17:05:40,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] FINISH, DestroyVDSCommand, log id: 9d52f7b
>2014-07-29 17:05:40,735 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 5e9a73e1
>2014-07-29 17:05:40,743 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Correlation ID: 59cf20f4, Job ID: f2a8e708-7284-4867-9a62-9941ca4445e6, Call Stack: null, Custom Event ID: -1, Message: VM shutdown initiated by admin on VM testtesttest (Host: rack) (Reason: Not Specified).
>2014-07-29 17:05:58,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 6cc96aca
>2014-07-29 17:05:58,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, DestroyVDSCommand, log id: 6cc96aca
>2014-07-29 17:05:59,014 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [72971053] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest is down. Exit message: User shut down from within the guest
>2014-07-29 17:05:59,017 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [72971053] VM testtesttest (78722009-2727-4715-91bf-dbbb33ce111e) is running in db and not running in VDS rack
>2014-07-29 17:05:59,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 447b6aa4
>2014-07-29 17:05:59,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, FullListVdsCommand, return: [], log id: 447b6aa4
>2014-07-29 17:05:59,039 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-15) [77cf3175] Running command: ProcessDownVmCommand internal: true.
>2014-07-29 17:07:27,287 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [9a2e425] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-29 17:07:27,324 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 17:07:27,335 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 2e1a74a
>2014-07-29 17:07:27,339 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] VDS rack is spm and moved from up calling resetIrs.
>2014-07-29 17:07:27,340 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 19813d4e
>2014-07-29 17:07:27,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4dec59c0
>2014-07-29 17:07:27,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-29 17:07:27,745 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, SpmStopVDSCommand, log id: 4dec59c0
>2014-07-29 17:07:27,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, ResetIrsVDSCommand, log id: 19813d4e
>2014-07-29 17:07:27,759 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, SetVdsStatusVDSCommand, log id: 2e1a74a
>2014-07-29 17:07:27,816 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-29 17:07:27,853 INFO  [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Running command: MaintenanceVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 17:07:27,864 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Correlation ID: 9a2e425, Job ID: a3fe84e8-7989-41b1-8645-4e4a8bc50e02, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin.
>2014-07-29 17:07:29,971 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack
>2014-07-29 17:07:29,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack.
>2014-07-29 17:07:29,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache
>2014-07-29 17:07:29,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache
>2014-07-29 17:07:30,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: 3833a4fb
>2014-07-29 17:07:33,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, DisconnectStoragePoolVDSCommand, log id: 3833a4fb
>2014-07-29 17:07:33,769 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 17:07:33,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 665d8683
>2014-07-29 17:07:33,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 665d8683
>2014-07-29 17:07:37,269 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts
>2014-07-29 17:07:37,275 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [65d9de58] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 17:07:37,282 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [65d9de58] Correlation ID: 65d9de58, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-07-29 17:15:45,527 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-11) [c4e13d9] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-29 17:15:45,554 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 17:15:45,556 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Before acquiring lock in order to prevent monitoring for host rack from data-center Default
>2014-07-29 17:15:45,557 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default
>2014-07-29 17:15:45,566 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6b5ec9da
>2014-07-29 17:15:45,572 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] FINISH, SetVdsStatusVDSCommand, log id: 6b5ec9da
>2014-07-29 17:15:45,580 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default
>2014-07-29 17:15:45,587 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Correlation ID: c4e13d9, Job ID: 7bc35737-4f02-4b77-a702-ebba530509e8, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin.
>2014-07-29 17:15:45,595 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-07-29 17:15:46,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5ccb869f
>2014-07-29 17:15:46,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 5ccb869f
>2014-07-29 17:15:46,620 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [ba69bba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 17:15:46,671 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [f062e4c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 17:15:46,678 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 17:15:46,689 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74d27179
>2014-07-29 17:15:46,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 74d27179
>2014-07-29 17:15:46,789 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] Host rack storage connection was succeeded 
>2014-07-29 17:15:46,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6384ea71
>2014-07-29 17:16:49,795 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, ConnectStoragePoolVDSCommand, log id: 6384ea71
>2014-07-29 17:16:49,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6a2dd8
>2014-07-29 17:16:49,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6a2dd8
>2014-07-29 17:16:49,910 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7cbab491] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-29 17:16:49,916 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7cbab491] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 17:16:49,939 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 17:16:58,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-29 17:16:58,146 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-29 17:16:58,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75df2592
>2014-07-29 17:16:58,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling started: taskId = 4680fed7-f396-4eaf-a348-8683a11b37ab
>2014-07-29 17:16:59,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling ended: taskId = 4680fed7-f396-4eaf-a348-8683a11b37ab task status = finished
>2014-07-29 17:16:59,261 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling ended, spm status: SPM
>2014-07-29 17:16:59,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-43) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4680fed7-f396-4eaf-a348-8683a11b37ab), log id: 7995adce
>2014-07-29 17:16:59,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, HSMClearTaskVDSCommand, log id: 7995adce
>2014-07-29 17:16:59,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d1df7c3, log id: 75df2592
>2014-07-29 17:16:59,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-29 17:16:59,319 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-29 17:16:59,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6b6c8d77
>2014-07-29 17:16:59,383 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-29 17:16:59,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 54df2b88
>2014-07-29 17:16:59,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 54df2b88
>2014-07-29 17:16:59,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b6c8d77
>2014-07-29 17:16:59,392 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default
>2014-07-29 17:19:08,248 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-07-29 17:19:08,253 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 17:19:08,255 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-07-29 17:23:26,834 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-07-29 17:23:26,844 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 17:23:26,845 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-07-29 17:23:54,993 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized
>2014-07-29 17:24:01,161 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf.
>2014-07-29 17:25:23,674 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-07-29 17:27:07,853 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-07-29 17:27:07,860 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-29 17:27:07,862 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-07-29 18:05:15,462 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-20) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-29 18:23:55,000 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized
>2014-07-29 18:24:01,189 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-07-29 18:30:45,659 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-22) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-29 19:23:55,005 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-07-29 19:24:01,218 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-07-29 20:23:55,012 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized
>2014-07-29 20:24:01,230 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Attempting to update VMs/Templates Ovf.
>2014-07-29 21:23:55,019 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized
>2014-07-29 21:24:01,244 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf.
>2014-07-29 22:23:55,027 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-07-29 22:24:01,270 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf.
>2014-07-29 22:45:46,731 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-29 22:47:13,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 22:47:13,176 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 22:47:13,179 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-55) [296dca52] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-29 22:47:13,192 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [296dca52] Correlation ID: 296dca52, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-29 22:49:13,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 22:49:13,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [296dca52] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 22:49:13,224 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-29 22:49:13,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-29 22:49:13,259 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-29 22:49:13,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 139b7b5b
>2014-07-29 22:49:13,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f9fb08c
>2014-07-29 22:49:13,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-29 22:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering 1 storage domains
>2014-07-29 22:50:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-29 22:50:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-29 22:50:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/29/14 10:50 PM
>2014-07-29 22:50:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-29 22:50:00,030 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 58a81b51
>2014-07-29 22:51:13,190 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-29 22:51:13,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] FINISH, SpmStopVDSCommand, log id: 5f9fb08c
>2014-07-29 22:51:13,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 22:51:13,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-64) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-29 22:51:13,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-29 22:51:13,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] FINISH, ResetIrsVDSCommand, log id: 139b7b5b
>2014-07-29 22:51:13,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] FINISH, ConnectStorageServerVDSCommand, log id: 58a81b51
>2014-07-29 22:51:13,206 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-29 22:51:13,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [7ef2d911] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-29 22:51:13,212 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-29 22:51:13,213 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-29 22:51:13,230 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [70fe0194] Correlation ID: 70fe0194, Job ID: 8663b09a-7f30-48d3-8746-f3236bd6df15, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-29 22:51:13,225 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-29 22:51:13,257 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/29/14 10:51 PM
>2014-07-29 22:51:13,262 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [296dca52] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 22:51:13,264 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-29 22:51:13,278 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-29 22:51:13,323 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-29 22:51:13,342 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [e344fc9] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-29 22:51:13,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [e344fc9] Removing vds [] from the domain in maintenance cache
>2014-07-29 22:51:13,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [e344fc9] Removing host(s) [] from hosts unseen domain report cache
>2014-07-29 22:51:13,350 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [296dca52] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-29 22:51:16,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 19ed538b
>2014-07-29 22:51:16,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, GetHardwareInfoVDSCommand, log id: 19ed538b
>2014-07-29 22:51:16,630 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 22:51:16,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3758779d
>2014-07-29 22:51:16,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] FINISH, DestroyVDSCommand, log id: 3758779d
>2014-07-29 22:51:16,782 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [5801dc91] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 22:51:16,789 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-29 22:51:16,795 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15c00120
>2014-07-29 22:51:16,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 15c00120
>2014-07-29 22:51:16,850 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] Host rack storage connection was succeeded 
>2014-07-29 22:51:16,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6892e72a
>2014-07-29 22:51:23,217 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-74) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-29 22:52:17,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 6892e72a
>2014-07-29 22:52:17,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7083f781
>2014-07-29 22:52:18,018 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7083f781
>2014-07-29 22:52:18,028 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [36555a0f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-29 22:52:18,034 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [36555a0f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-29 22:52:18,061 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-29 22:52:23,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-29 22:52:23,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-29 22:52:23,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a4b5b4e
>2014-07-29 22:52:23,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling started: taskId = 7961f177-aa71-4305-ad73-4952dd174589
>2014-07-29 22:52:26,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling ended: taskId = 7961f177-aa71-4305-ad73-4952dd174589 task status = finished
>2014-07-29 22:52:26,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling ended, spm status: SPM
>2014-07-29 22:52:26,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7961f177-aa71-4305-ad73-4952dd174589), log id: 45f0eee3
>2014-07-29 22:52:26,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, HSMClearTaskVDSCommand, log id: 45f0eee3
>2014-07-29 22:52:26,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@394ceed4, log id: 6a4b5b4e
>2014-07-29 22:52:26,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-29 22:52:26,704 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-29 22:52:26,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 34d9648
>2014-07-29 22:52:27,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-29 22:52:27,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e760c
>2014-07-29 22:52:27,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e760c
>2014-07-29 22:52:27,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34d9648
>2014-07-29 22:52:27,291 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-9) Discovered no tasks on Storage Pool Default
>2014-07-29 23:01:14,482 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4dca0641] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.85743 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-29 23:23:55,036 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized
>2014-07-29 23:24:01,284 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-07-29 23:36:53,513 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-4) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 00:23:55,044 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) Principal admin::internal synchronized
>2014-07-30 00:24:01,314 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-07-30 01:23:55,050 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized
>2014-07-30 01:24:01,348 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
>2014-07-30 02:23:55,056 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized
>2014-07-30 02:24:01,382 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf.
>2014-07-30 03:23:55,062 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [4494a716] Principal admin::internal synchronized
>2014-07-30 03:24:01,394 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf.
>2014-07-30 03:35:35,000 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Start deleteAgedOutAuditLogs
>2014-07-30 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-14) Start deleteAgedOutCommandEntities
>2014-07-30 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-14) Finished deleteAgedOutCommandEntities
>2014-07-30 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Finished deleteAgedOutAuditLogs
>2014-07-30 04:23:55,069 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized
>2014-07-30 04:24:01,409 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-07-30 05:01:16,631 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 05:02:52,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 05:02:52,618 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 05:02:52,621 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 05:02:52,638 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Correlation ID: 29bdffa0, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 05:04:52,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 05:04:52,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [29bdffa0] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 05:04:52,667 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-30 05:04:52,669 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 05:04:52,705 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 05:04:52,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: b4ea484
>2014-07-30 05:04:52,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4920ce7b
>2014-07-30 05:04:52,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 05:05:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) Autorecovering 1 storage domains
>2014-07-30 05:05:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-30 05:05:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 05:05:00,015 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/30/14 5:05 AM
>2014-07-30 05:05:00,027 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 05:05:00,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 465107e2
>2014-07-30 05:06:00,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 465107e2
>2014-07-30 05:06:00,048 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/30/14 5:06 AM
>2014-07-30 05:06:52,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 05:06:52,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 05:06:52,636 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-30 05:06:52,638 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 05:06:52,643 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [2e41028] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-30 05:06:52,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] FINISH, SpmStopVDSCommand, log id: 4920ce7b
>2014-07-30 05:06:52,646 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [2e41028] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 05:06:52,648 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [2e41028] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-30 05:06:52,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] FINISH, ResetIrsVDSCommand, log id: b4ea484
>2014-07-30 05:06:52,680 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [2e41028] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-30 05:06:52,687 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [47196bb9] Correlation ID: 47196bb9, Job ID: da38ecad-2a3a-4d07-8f38-fab75caaf397, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 05:06:52,694 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [29bdffa0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 05:06:52,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 05:06:52,760 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 05:06:52,779 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [1f774652] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-30 05:06:52,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [1f774652] Removing vds [] from the domain in maintenance cache
>2014-07-30 05:06:52,787 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [1f774652] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 05:06:52,788 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [29bdffa0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 05:06:55,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 19cb0b01
>2014-07-30 05:06:56,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] FINISH, GetHardwareInfoVDSCommand, log id: 19cb0b01
>2014-07-30 05:06:56,036 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [27efeab7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 05:06:56,086 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [15769bf8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 05:06:56,094 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 05:06:56,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1589ef68
>2014-07-30 05:06:56,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 1589ef68
>2014-07-30 05:06:56,166 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] Host rack storage connection was succeeded 
>2014-07-30 05:06:56,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 326756f0
>2014-07-30 05:07:02,650 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-27) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 05:07:56,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 326756f0
>2014-07-30 05:07:56,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 704871c5
>2014-07-30 05:07:56,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] FINISH, SetMOMPolicyParametersVDSCommand, log id: 704871c5
>2014-07-30 05:07:56,650 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7ec29012] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-30 05:07:56,666 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7ec29012] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-30 05:07:56,683 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 05:08:02,796 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 05:08:02,802 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 05:08:02,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72f55fb4
>2014-07-30 05:08:02,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling started: taskId = 22041af6-af8d-4930-bc1f-e69d1cdad8c5
>2014-07-30 05:08:04,863 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling ended: taskId = 22041af6-af8d-4930-bc1f-e69d1cdad8c5 task status = finished
>2014-07-30 05:08:04,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling ended, spm status: SPM
>2014-07-30 05:08:04,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=22041af6-af8d-4930-bc1f-e69d1cdad8c5), log id: 5ae2832f
>2014-07-30 05:08:04,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] FINISH, HSMClearTaskVDSCommand, log id: 5ae2832f
>2014-07-30 05:08:04,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1739ea18, log id: 72f55fb4
>2014-07-30 05:08:04,971 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 05:08:04,978 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [2734597c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 05:08:04,980 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-20) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 05:08:04,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3d9f180f
>2014-07-30 05:08:05,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 05:08:05,014 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 14aaf8ec
>2014-07-30 05:08:05,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14aaf8ec
>2014-07-30 05:08:05,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d9f180f
>2014-07-30 05:08:05,022 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [2734597c] Discovered no tasks on Storage Pool Default
>2014-07-30 05:23:55,076 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-07-30 05:24:01,424 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf.
>2014-07-30 06:23:55,085 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized
>2014-07-30 06:24:01,437 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-07-30 07:23:55,091 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized
>2014-07-30 07:24:01,451 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf.
>2014-07-30 08:23:55,096 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized
>2014-07-30 08:24:01,466 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
>2014-07-30 08:29:38,503 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 09:18:13,768 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-07-30 09:18:13,773 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 09:18:19,166 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [48432f1c] Running command: LogoutUserCommand internal: false.
>2014-07-30 09:18:19,174 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [48432f1c] Correlation ID: 48432f1c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-30 09:23:55,102 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-28) [8ef1638] Principal admin::internal synchronized
>2014-07-30 09:24:01,481 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf.
>2014-07-30 09:31:14,716 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.12679 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-30 10:23:55,110 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Principal admin::internal synchronized
>2014-07-30 10:24:01,494 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf.
>2014-07-30 10:41:22,810 INFO  [org.ovirt.engine.core.bll.LoginBaseCommand] (ajp--127.0.0.1-8702-3) Cant login user "admin" with authentication profile "internal" because the authentication failed.
>2014-07-30 10:41:22,817 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin cannot login, please verify the username and password.
>2014-07-30 10:41:22,821 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin failed to log in.
>2014-07-30 10:41:22,822 WARN  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) CanDoAction of action LoginAdminUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD
>2014-07-30 10:41:27,478 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
>2014-07-30 10:41:27,482 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 10:41:27,527 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-30 10:42:31,374 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.204:/home/iso/shared value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 10:42:31,386 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 10:42:31,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [29002c1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3370d1f8
>2014-07-30 10:42:31,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [29002c1] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3370d1f8
>2014-07-30 10:42:31,474 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.204:/home/iso/shared value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 10:42:31,560 INFO  [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-7) START, GetExistingStorageDomainListQuery(refresh: true, filtered: false), log id: 286ff6cf
>2014-07-30 10:42:31,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-7) START, HSMGetStorageDomainsListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ISO, path=10.34.63.204:/home/iso/shared), log id: 668150fa
>2014-07-30 10:43:32,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, HSMGetStorageDomainsListVDSCommand, return: [0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71], log id: 668150fa
>2014-07-30 10:43:32,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-7) START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 70465c83
>2014-07-30 10:43:32,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@440d84d6, 00000002-0002-0002-0002-0000000000a8>, log id: 70465c83
>2014-07-30 10:43:32,177 INFO  [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-7) FINISH, GetExistingStorageDomainListQuery, log id: 286ff6cf
>2014-07-30 10:43:37,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, HSMGetStorageDomainsListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ISO, path=), log id: b0b433b
>2014-07-30 10:44:37,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, HSMGetStorageDomainsListVDSCommand, return: [0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71], log id: b0b433b
>2014-07-30 10:44:37,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 7ae22fb8
>2014-07-30 10:44:37,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@440d84d6, 00000002-0002-0002-0002-0000000000a8>, log id: 7ae22fb8
>2014-07-30 10:44:37,622 INFO  [org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand] (ajp--127.0.0.1-8702-4) [1729b243] Running command: AddExistingFileStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 10:44:37,629 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 6ca5df6b
>2014-07-30 10:44:37,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@56c210af, log id: 6ca5df6b
>2014-07-30 10:44:37,692 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [1729b243] Correlation ID: 1729b243, Job ID: 5904ce5c-95d8-4c08-a50b-4df2b86c6343, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was added by admin
>2014-07-30 10:44:37,870 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-2) [26e69bac] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 10:44:37,883 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 10:44:37,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75cd4335
>2014-07-30 10:44:37,902 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 75cd4335
>2014-07-30 10:44:37,905 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 1b2b0aad
>2014-07-30 10:44:41,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] FINISH, AttachStorageDomainVDSCommand, log id: 1b2b0aad
>2014-07-30 10:44:41,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 694af1c2
>2014-07-30 10:44:41,365 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] FINISH, GetImagesListVDSCommand, return: [], log id: 694af1c2
>2014-07-30 10:44:41,380 WARN  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-2) There are no OVF_STORE disks on storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71
>2014-07-30 10:44:41,385 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 10:44:41,403 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 10:44:41,409 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 10:44:41,410 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 10:44 AM
>2014-07-30 10:44:41,415 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 10:44:41,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 61e452eb
>2014-07-30 10:44:41,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 61e452eb
>2014-07-30 10:44:41,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [6ef20677] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 3a04519a
>2014-07-30 10:44:41,596 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [6ef20677] FINISH, ActivateStorageDomainVDSCommand, log id: 3a04519a
>2014-07-30 10:44:41,597 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 10:44 AM
>2014-07-30 10:44:41,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 409841cf
>2014-07-30 10:44:56,319 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Lock Acquired to object EngineLock [exclusiveLocks= key: ovirtnode value: VM_NAME
>, sharedLocks= ]
>2014-07-30 10:44:56,379 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Running command: AddVmFromScratchCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-07-30 10:44:56,522 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [78d048a] Correlation ID: 78d048a, Job ID: 804cbe8f-16a6-42ef-b810-2fef73e7462a, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was created by admin.
>2014-07-30 10:44:56,528 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Lock freed to object EngineLock [exclusiveLocks= key: ovirtnode value: VM_NAME
>, sharedLocks= ]
>2014-07-30 10:44:57,280 INFO  [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-7) [5e88f035] Running command: AddVmInterfaceCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM,  ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile
>2014-07-30 10:44:57,319 INFO  [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-7) [7c1ea83c] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:44:57,347 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [7c1ea83c] Correlation ID: 7c1ea83c, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM ovirtnode. (User: admin)
>2014-07-30 10:44:57,364 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [7c1ea83c] Correlation ID: 5e88f035, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM ovirtnode. (User: admin)
>2014-07-30 10:44:57,441 INFO  [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-3) [17a8e6f6] Running command: ReorderVmNicsCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:45:04,813 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-11) [2126a430] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM_DISK_BOOT
>, sharedLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>]
>2014-07-30 10:45:04,834 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-11) [2126a430] Running command: AddDiskCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 10:45:04,879 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 10:45:04,894 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM_DISK_BOOT
>, sharedLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>]
>2014-07-30 10:45:04,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageSizeInBytes = 10737418240, volumeFormat = RAW, newImageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614, newImageDescription = ), log id: 4b17b6f0
>2014-07-30 10:45:04,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID
>2014-07-30 10:45:05,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] FINISH, CreateImageVDSCommand, return: 1efc6019-96fd-4b59-8289-6ce7fe14a614, log id: 4b17b6f0
>2014-07-30 10:45:05,407 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-11) [1f7f6bf1] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf
>2014-07-30 10:45:05,408 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-11) [1f7f6bf1] CommandMultiAsyncTasks::AttachTask: Attaching task 01677897-7f42-4cd2-8303-56109bb9ef14 to command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf.
>2014-07-30 10:45:05,417 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Adding task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-30 10:45:05,436 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Correlation ID: 2126a430, Job ID: d5224cf3-594f-4add-b92f-f1e82f96e8fd, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of ovirtnode_Disk1 was initiated on VM ovirtnode by admin.
>2014-07-30 10:45:05,437 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-11) [1f7f6bf1] BaseAsyncTask::startPollingTask: Starting to poll task 01677897-7f42-4cd2-8303-56109bb9ef14.
>2014-07-30 10:45:13,116 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-43) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-07-30 10:45:13,123 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-43) SPMAsyncTask::PollTask: Polling task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-30 10:45:13,130 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-43) BaseAsyncTask::OnTaskEndSuccess: Task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-30 10:45:13,132 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-43) CommandAsyncTask::EndActionIfNecessary: All tasks of command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf has ended -> executing endAction
>2014-07-30 10:45:13,133 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-43) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: fa8dcdeb-d342-45d0-97e9-6acc112fe2cf): calling endAction .
>2014-07-30 10:45:13,135 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0
>2014-07-30 10:45:13,146 INFO  [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-23) [2126a430] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand
>2014-07-30 10:45:13,152 INFO  [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand
>2014-07-30 10:45:13,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614), log id: 707002a5
>2014-07-30 10:45:13,184 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8fd12089, log id: 707002a5
>2014-07-30 10:45:13,201 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: 2126a430, Call Stack: null, Custom Event ID: -1, Message: The disk ovirtnode_Disk1 was successfully added to VM ovirtnode.
>2014-07-30 10:45:13,202 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result.
>2014-07-30 10:45:13,203 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks.
>2014-07-30 10:45:13,207 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-23) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 01677897-7f42-4cd2-8303-56109bb9ef14
>2014-07-30 10:45:13,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 01677897-7f42-4cd2-8303-56109bb9ef14), log id: 1b8d2334
>2014-07-30 10:45:13,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=01677897-7f42-4cd2-8303-56109bb9ef14), log id: 4f02390d
>2014-07-30 10:45:13,267 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, HSMClearTaskVDSCommand, log id: 4f02390d
>2014-07-30 10:45:13,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, SPMClearTaskVDSCommand, log id: 1b8d2334
>2014-07-30 10:45:13,275 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-23) BaseAsyncTask::RemoveTaskFromDB: Removed task 01677897-7f42-4cd2-8303-56109bb9ef14 from DataBase
>2014-07-30 10:45:13,276 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity fa8dcdeb-d342-45d0-97e9-6acc112fe2cf
>2014-07-30 10:45:16,487 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 10:45:16,528 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: da8cb13
>2014-07-30 10:45:16,529 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: da8cb13
>2014-07-30 10:45:16,586 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Running command: RunVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:45:16,622 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@3f3f6d6b), log id: 4107216f
>2014-07-30 10:45:16,637 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4107216f
>2014-07-30 10:45:16,659 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 410e0697
>2014-07-30 10:45:16,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 40951fe9
>2014-07-30 10:45:16,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs
>CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog
>ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0
>Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiMTRhZjMwMDYtMmJhMC00YWJj
>LThlNGQtMjNmNjA5MGMyY2U1IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz
>ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ==
>, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg
>YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp
>cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0
>Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg
>IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK
>, openstack/content/0000=YXV0bwo=
>}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=58b5c483-7135-422b-81ae-ef1624176b93}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=cbd5ab2a-dc25-451c-9407-ab762c3570da}, {shared=false, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, index=0, iface=virtio, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true
>2014-07-30 10:45:16,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, CreateVmFromCloudInitVDSCommand, log id: 40951fe9
>2014-07-30 10:45:16,788 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 410e0697
>2014-07-30 10:45:16,790 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 10:45:16,795 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Correlation ID: 6026cfc8, Job ID: 096197b2-7ed6-4ea0-ad29-fd919dca24af, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack).
>2014-07-30 10:45:41,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 409841cf
>2014-07-30 10:45:41,892 INFO  [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-1) Refreshed vds rack in pool Default
>2014-07-30 10:45:41,894 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 10:45 AM
>2014-07-30 10:45:41,896 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-29) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 10:45:41,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 26fc1628
>2014-07-30 10:45:41,904 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6ef20677] Correlation ID: 6ef20677, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was activated by Non interactive user
>2014-07-30 10:45:41,919 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6ef20677] Correlation ID: 26e69bac, Job ID: 747052ef-c6f7-4a4e-8e32-2682e514cb99, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was attached to Data Center Default by admin
>2014-07-30 10:45:42,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 26fc1628
>2014-07-30 10:45:42,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: df24d4
>2014-07-30 10:45:42,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: df24d4
>2014-07-30 10:45:44,923 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp
>2014-07-30 10:45:44,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-65) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 63f38ad7
>2014-07-30 10:45:44,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=22803, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@a205c85, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 63f38ad7
>2014-07-30 10:45:44,946 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-30 10:45:44,947 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-30 10:45:47,666 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:45:47,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, ticket=+2QnNV2WQwoy, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 269088d1
>2014-07-30 10:45:47,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] FINISH, SetVmTicketVDSCommand, log id: 269088d1
>2014-07-30 10:45:47,799 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4bc8cf97] Correlation ID: 4bc8cf97, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM ovirtnode
>2014-07-30 10:45:54,200 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM ovirtnode.
>2014-07-30 10:46:12,468 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM ovirtnode.
>2014-07-30 10:46:16,115 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] Running command: StopVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:46:16,120 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 289cd342
>2014-07-30 10:46:16,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 7c3ef896
>2014-07-30 10:46:16,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] FINISH, DestroyVDSCommand, log id: 7c3ef896
>2014-07-30 10:46:16,573 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] FINISH, DestroyVmVDSCommand, return: Down, log id: 289cd342
>2014-07-30 10:46:16,580 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) [636d32ad] Correlation ID: 636d32ad, Job ID: 6ec8048d-f065-4954-a4c6-ca220f2e9e26, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified).
>2014-07-30 10:46:16,586 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-36) [12ece3d4] Running command: ProcessDownVmCommand internal: true.
>2014-07-30 10:46:29,059 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Setting new tasks map. The map contains now 0 tasks
>2014-07-30 10:46:29,061 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-30 10:47:02,486 INFO  [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) [31cbf1b] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 10:47:02,612 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 7a3461a9
>2014-07-30 10:47:02,614 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7a3461a9
>2014-07-30 10:47:02,655 INFO  [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) Running command: RunVmOnceCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:47:02,683 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) Running VM with attached cd ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso
>2014-07-30 10:47:02,686 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@66810750), log id: 6401d514
>2014-07-30 10:47:02,690 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, UpdateVmDynamicDataVDSCommand, log id: 6401d514
>2014-07-30 10:47:02,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (ajp--127.0.0.1-8702-4) START, IsoPrefixVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000002-0002-0002-0002-0000000001d9), log id: 49805003
>2014-07-30 10:47:02,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/10.34.63.204:_home_iso_shared/0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71/images/11111111-1111-1111-1111-111111111111, log id: 49805003
>2014-07-30 10:47:02,728 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 761cd6f4
>2014-07-30 10:47:02,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 8bea027
>2014-07-30 10:47:02,776 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, bootOrder=2, iface=ide, index=2, specParams={}, path=/rhev/data-center/mnt/10.34.63.204:_home_iso_shared/0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71/images/11111111-1111-1111-1111-111111111111/ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso, device=cdrom, type=disk, readonly=true, deviceId=4329cc23-372a-4399-a974-a87ca8cd3d0d}, {shared=false, index=0, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, propagateErrors=off, format=raw, type=disk, bootOrder=1, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {bootOrder=3, nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true
>2014-07-30 10:47:02,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, CreateVDSCommand, log id: 8bea027
>2014-07-30 10:47:02,812 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 761cd6f4
>2014-07-30 10:47:02,813 INFO  [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 10:47:02,824 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 31cbf1b, Job ID: 4a7ed238-134e-41ad-98e9-6748c617f590, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack).
>2014-07-30 10:47:03,941 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp
>2014-07-30 10:47:03,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 31d148f5
>2014-07-30 10:47:03,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=23146, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@230ce660, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 31d148f5
>2014-07-30 10:47:03,957 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-30 10:47:03,959 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-30 10:47:17,404 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [ff43ced] Running command: SetVmTicketCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:47:17,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [ff43ced] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, ticket=LQf3rGZKSh2M, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 160a92aa
>2014-07-30 10:47:17,443 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [ff43ced] FINISH, SetVmTicketVDSCommand, log id: 160a92aa
>2014-07-30 10:47:17,453 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [ff43ced] Correlation ID: ff43ced, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM ovirtnode
>2014-07-30 10:47:22,211 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [375fcc6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM ovirtnode.
>2014-07-30 10:48:04,665 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from PoweringUp --> Up
>2014-07-30 10:48:04,678 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: 31cbf1b, Job ID: 4a7ed238-134e-41ad-98e9-6748c617f590, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack
>2014-07-30 10:50:43,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Failed in SpmStatusVDS method
>2014-07-30 10:50:43,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = (-202, 'Sanlock resource read failure', 'Sanlock exception'), code = 100
>2014-07-30 10:50:43,088 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [52a75f18] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 10:50:43,092 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [52a75f18] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 10:50:43,094 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [52a75f18] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 status from Active to Unknown, reason : null
>2014-07-30 10:50:43,106 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52a75f18] Correlation ID: 52a75f18, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: General Exception).
>2014-07-30 10:50:45,050 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 10:50:49,121 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [52a75f18] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-07-30 10:50:49,127 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [52a75f18] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 10:50:49,134 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52a75f18] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 10:50:49,136 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-34) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 10:50:49,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 638a5d14
>2014-07-30 10:50:51,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 10:50:51,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45200898
>2014-07-30 10:50:51,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 45200898
>2014-07-30 10:50:51,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 638a5d14
>2014-07-30 10:50:51,724 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-34) [52a75f18] Discovered no tasks on Storage Pool Default
>2014-07-30 10:51:00,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-07-30 10:51:00,170 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-07-30 10:52:30,878 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> RebootInProgress
>2014-07-30 10:52:30,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-60) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 5b6e6aa3
>2014-07-30 10:52:30,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=-1, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=23146, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=10.34.130.128, devices=[Ljava.lang.Object;@2e1b223a, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 5b6e6aa3
>2014-07-30 10:52:30,894 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-30 10:52:30,896 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-30 10:53:19,480 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from RebootInProgress --> Up
>2014-07-30 10:58:49,226 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM ovirtnode.
>2014-07-30 10:58:52,573 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] Running command: StopVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 10:58:52,578 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 21fcde55
>2014-07-30 10:58:52,584 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3f7d478f
>2014-07-30 10:58:53,466 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] FINISH, DestroyVDSCommand, log id: 3f7d478f
>2014-07-30 10:58:53,472 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] FINISH, DestroyVmVDSCommand, return: Down, log id: 21fcde55
>2014-07-30 10:58:53,483 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [536a6748] Correlation ID: 536a6748, Job ID: fa6da149-a69e-4562-b193-07632374556a, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified).
>2014-07-30 10:58:53,530 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-20) [13562194] Running command: ProcessDownVmCommand internal: true.
>2014-07-30 11:00:35,989 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [a757eeb] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 11:00:36,070 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [a757eeb] START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 5e2cf264
>2014-07-30 11:00:36,071 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [a757eeb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5e2cf264
>2014-07-30 11:00:36,109 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Running command: RunVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-07-30 11:00:36,132 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@823b1cc3), log id: 2578f29c
>2014-07-30 11:00:36,136 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2578f29c
>2014-07-30 11:00:36,141 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: e68d50e
>2014-07-30 11:00:36,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 54c3a16
>2014-07-30 11:00:36,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=cbd5ab2a-dc25-451c-9407-ab762c3570da}, {shared=false, index=0, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true
>2014-07-30 11:00:36,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, CreateVDSCommand, log id: 54c3a16
>2014-07-30 11:00:36,202 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: e68d50e
>2014-07-30 11:00:36,203 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-07-30 11:00:36,209 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Correlation ID: a757eeb, Job ID: dcca760f-4b55-4584-bc7d-fb073e231cd0, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack).
>2014-07-30 11:00:38,351 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp
>2014-07-30 11:00:38,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-89) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 580a0f2d
>2014-07-30 11:00:38,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=24451, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@7c6a8082, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 580a0f2d
>2014-07-30 11:00:38,374 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-07-30 11:00:38,375 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics}
>2014-07-30 11:01:39,046 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from PoweringUp --> Up
>2014-07-30 11:01:39,060 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: a757eeb, Job ID: dcca760f-4b55-4584-bc7d-fb073e231cd0, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack
>2014-07-30 11:10:10,427 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared in problem. vds: rack
>2014-07-30 11:10:25,553 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 11:13:42,106 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> NotResponding
>2014-07-30 11:13:42,167 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode is not responding.
>2014-07-30 11:13:45,220 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from NotResponding --> Up
>2014-07-30 11:15:10,430 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared
>2014-07-30 11:15:10,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-30 11:15:10,508 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 11:15:10,515 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 11:15:10,524 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 11:15:10,542 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Correlation ID: d7a3c40, Job ID: 4781ad59-ef44-4d90-b7a2-63660226cd8f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-30 11:15:10,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Removing vds [] from the domain in maintenance cache
>2014-07-30 11:15:10,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 11:15:20,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 11:15:20,795 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 11:15:20,798 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 11:15:20,807 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: 52d379dc, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 11:15:25,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 11:15:25,561 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 11:15:25,591 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 11:15:25,621 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3b907c7a
>2014-07-30 11:15:25,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4d1ddc64
>2014-07-30 11:15:25,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 11:18:20,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 11:18:20,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 11:18:20,839 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-30 11:18:20,840 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 11:18:25,637 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-07-30 11:18:25,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] FINISH, SpmStopVDSCommand, log id: 4d1ddc64
>2014-07-30 11:18:25,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] FINISH, ResetIrsVDSCommand, log id: 3b907c7a
>2014-07-30 11:18:25,662 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Correlation ID: 7aeae3d7, Job ID: 371e7693-0331-4087-9db8-836d853059d5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 11:18:25,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Removing vds [] from the domain in maintenance cache
>2014-07-30 11:18:25,670 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 11:18:25,671 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 11:18:25,672 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] The following operation Event [eventType=RECONSTRUCT, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, vdsId=null, description=IrsBrokerCommand.startReconstruct()] was cancelled, because of reconstruct was run before
>2014-07-30 11:18:25,674 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] The following operation Event [eventType=DOMAINMONITORING, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=null, vdsId=17f7fe56-5f32-40ef-b098-611b84484e0b, description=] was cancelled, because of reconstruct was run before
>2014-07-30 11:19:20,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 11:19:20,804 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 11:19:23,257 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> Paused
>2014-07-30 11:19:30,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool Default
>2014-07-30 11:19:30,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 446581d7
>2014-07-30 11:19:41,500 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by Host rack as Active in Pool 00000002-0002-0002-0002-0000000001d9, moving to active status
>2014-07-30 11:19:41,506 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by Host rack as Active in Pool 00000002-0002-0002-0002-0000000001d9, moving to active status
>2014-07-30 11:20:31,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, ConnectStoragePoolVDSCommand, log id: 446581d7
>2014-07-30 11:20:31,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 11:20:31,420 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 11:20:31,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 237983e6
>2014-07-30 11:20:31,472 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling started: taskId = 7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4
>2014-07-30 11:20:32,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling ended: taskId = 7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4 task status = finished
>2014-07-30 11:20:32,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling ended, spm status: SPM
>2014-07-30 11:20:32,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4), log id: 510712fd
>2014-07-30 11:20:32,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, HSMClearTaskVDSCommand, log id: 510712fd
>2014-07-30 11:20:32,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@417c9d1, log id: 237983e6
>2014-07-30 11:20:32,589 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 11:20:32,596 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 11:20:32,598 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 11:20:32,616 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 71ab0f30
>2014-07-30 11:20:32,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 11:20:32,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3c5a4d50
>2014-07-30 11:20:32,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c5a4d50
>2014-07-30 11:20:32,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 71ab0f30
>2014-07-30 11:20:32,642 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-21) Discovered no tasks on Storage Pool Default
>2014-07-30 11:23:55,117 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized
>2014-07-30 11:24:01,527 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf.
>2014-07-30 11:24:01,673 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-30 11:24:01,682 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 11:24:01,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 56c3f1a1
>2014-07-30 11:24:01,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-30 11:24:01,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014"}
>
>2014-07-30 11:24:04,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] FINISH, SetVolumeDescriptionVDSCommand, log id: 56c3f1a1
>2014-07-30 11:24:04,550 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 11:24:04,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7268035d
>2014-07-30 11:24:04,553 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeVdsBrokerCommand: 
>2014-07-30 11:24:04,554 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                size=20,480
>
>2014-07-30 11:24:20,251 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, UploadStreamVDSCommand, log id: 7268035d
>2014-07-30 11:24:20,259 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) [233b68fb] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 03335b91-a86c-46c6-8274-c21b3b5a9179
>2014-07-30 11:24:20,261 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-46) [233b68fb] CommandMultiAsyncTasks::AttachTask: Attaching task 13cca31c-d228-4247-a3cd-c324ae05433e to command 03335b91-a86c-46c6-8274-c21b3b5a9179.
>2014-07-30 11:24:20,274 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) [233b68fb] Adding task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-30 11:24:20,277 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-30 11:24:20,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 428e56f1
>2014-07-30 11:24:20,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-30 11:24:20,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014","Size":20480}
>
>2014-07-30 11:24:20,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, SetVolumeDescriptionVDSCommand, log id: 428e56f1
>2014-07-30 11:24:20,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 5804e57a
>2014-07-30 11:24:20,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-30 11:24:20,610 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014"}
>
>2014-07-30 11:24:20,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, SetVolumeDescriptionVDSCommand, log id: 5804e57a
>2014-07-30 11:24:20,988 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 11:24:20,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7572bf6
>2014-07-30 11:24:20,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- executeVdsBrokerCommand: 
>2014-07-30 11:24:20,992 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                size=20,480
>
>2014-07-30 11:24:21,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] FINISH, UploadStreamVDSCommand, log id: 7572bf6
>2014-07-30 11:24:21,848 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-46) [56f1581c] CommandMultiAsyncTasks::AttachTask: Attaching task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 to command 03335b91-a86c-46c6-8274-c21b3b5a9179.
>2014-07-30 11:24:21,854 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Adding task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-07-30 11:24:21,856 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-07-30 11:24:21,857 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-07-30 11:24:21,858 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-07-30 11:24:21,859 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-07-30 11:24:21,860 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-30 11:24:21,864 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 7976e0fd
>2014-07-30 11:24:21,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-07-30 11:24:21,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014","Size":20480}
>
>2014-07-30 11:24:21,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] FINISH, SetVolumeDescriptionVDSCommand, log id: 7976e0fd
>2014-07-30 11:24:21,994 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) [56f1581c] BaseAsyncTask::startPollingTask: Starting to poll task 13cca31c-d228-4247-a3cd-c324ae05433e.
>2014-07-30 11:24:21,995 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) [56f1581c] BaseAsyncTask::startPollingTask: Starting to poll task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559.
>2014-07-30 11:24:23,212 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now
>2014-07-30 11:24:23,218 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-30 11:24:23,220 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-30 11:24:23,222 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-49) Task with DB Task ID 0bb51d5e-c935-4fef-88aa-92f818eb4a10 and VDSM Task ID 13cca31c-d228-4247-a3cd-c324ae05433e is in state Polling. End action for command 03335b91-a86c-46c6-8274-c21b3b5a9179 will proceed when all the entitys tasks are completed.
>2014-07-30 11:24:23,224 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-07-30 11:24:23,226 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-07-30 11:24:23,227 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::EndActionIfNecessary: All tasks of command 03335b91-a86c-46c6-8274-c21b3b5a9179 has ended -> executing endAction
>2014-07-30 11:24:23,228 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 03335b91-a86c-46c6-8274-c21b3b5a9179): calling endAction .
>2014-07-30 11:24:23,230 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0
>2014-07-30 11:24:23,239 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand
>2014-07-30 11:24:23,242 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-07-30 11:24:23,244 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-07-30 11:24:23,246 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-07-30 11:24:23,247 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-07-30 11:24:23,248 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-07-30 11:24:23,250 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result.
>2014-07-30 11:24:23,252 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks.
>2014-07-30 11:24:23,253 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559
>2014-07-30 11:24:23,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 57ab43e6-b1ba-4a76-97af-26b0ed6e5559), log id: 6ca7ec2
>2014-07-30 11:24:23,259 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=57ab43e6-b1ba-4a76-97af-26b0ed6e5559), log id: 5847f609
>2014-07-30 11:24:23,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, HSMClearTaskVDSCommand, log id: 5847f609
>2014-07-30 11:24:23,299 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, SPMClearTaskVDSCommand, log id: 6ca7ec2
>2014-07-30 11:24:23,305 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] BaseAsyncTask::RemoveTaskFromDB: Removed task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 from DataBase
>2014-07-30 11:24:23,306 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 13cca31c-d228-4247-a3cd-c324ae05433e
>2014-07-30 11:24:23,307 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 13cca31c-d228-4247-a3cd-c324ae05433e), log id: 3b47f7bf
>2014-07-30 11:24:23,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=13cca31c-d228-4247-a3cd-c324ae05433e), log id: 71da3b00
>2014-07-30 11:24:23,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, HSMClearTaskVDSCommand, log id: 71da3b00
>2014-07-30 11:24:23,379 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, SPMClearTaskVDSCommand, log id: 3b47f7bf
>2014-07-30 11:24:23,385 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] BaseAsyncTask::RemoveTaskFromDB: Removed task 13cca31c-d228-4247-a3cd-c324ae05433e from DataBase
>2014-07-30 11:24:23,386 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 03335b91-a86c-46c6-8274-c21b3b5a9179
>2014-07-30 11:25:29,087 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-6) [4494a716] Setting new tasks map. The map contains now 0 tasks
>2014-07-30 11:25:29,089 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-6) [4494a716] Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-07-30 12:23:55,123 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized
>2014-07-30 12:24:21,996 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Attempting to update VMs/Templates Ovf.
>2014-07-30 12:28:54,585 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 12:28:54,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 523c8e76
>2014-07-30 12:28:54,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21656ec0
>2014-07-30 12:28:54,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: 523c8e76
>2014-07-30 12:28:54,912 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] Finished automatic refresh process for Floppy file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 12:28:55,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 21656ec0
>2014-07-30 12:28:55,687 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] Finished automatic refresh process for ISO file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 13:23:55,131 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) [2e742a17] Principal admin::internal synchronized
>2014-07-30 13:24:22,023 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf.
>2014-07-30 13:28:55,697 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 13:28:55,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 148e385
>2014-07-30 13:28:55,705 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 135c372a
>2014-07-30 13:28:56,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 148e385
>2014-07-30 13:28:56,549 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 13:28:56,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: 135c372a
>2014-07-30 13:28:56,714 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-47) Finished automatic refresh process for Floppy file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 13:44:54,175 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.54911 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator.
>2014-07-30 13:57:42,073 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-17) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 14:23:55,137 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized
>2014-07-30 14:24:22,059 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf.
>2014-07-30 14:25:13,526 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 14:25:13,529 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared in problem. vds: rack
>2014-07-30 14:28:56,724 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 14:28:56,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21a51a33
>2014-07-30 14:28:56,731 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e6a9841
>2014-07-30 14:30:13,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) starting processDomainRecovery for domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared
>2014-07-30 14:30:13,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-30 14:30:13,627 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 14:30:13,637 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 14:30:13,651 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 14:30:13,665 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Correlation ID: 6c3a9186, Job ID: abc6e47b-f0ed-4874-9b4b-8b6d316d0d5f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-30 14:30:13,673 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Removing vds [] from the domain in maintenance cache
>2014-07-30 14:30:13,674 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 14:30:13,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 14:30:13,681 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 14:30:13,704 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 14:30:13,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: a93288e
>2014-07-30 14:30:13,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 11255cc8
>2014-07-30 14:30:13,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 14:30:23,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 14:30:23,463 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 14:30:23,480 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Correlation ID: 2b0f39f8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 14:30:23,527 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]]
>
>2014-07-30 14:30:23,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] HostName = rack
>2014-07-30 14:30:23,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM
>2014-07-30 14:32:23,653 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-30 14:32:23,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 14:32:23,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] FINISH, SpmStopVDSCommand, log id: 11255cc8
>2014-07-30 14:32:23,658 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 14:32:23,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] FINISH, ResetIrsVDSCommand, log id: a93288e
>2014-07-30 14:32:23,670 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 14:32:23,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 14:32:23,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] FINISH, GetFileStatsVDSCommand, log id: 2e6a9841
>2014-07-30 14:32:23,675 WARN  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] The refresh process from VDSM, for Iso files failed.
>2014-07-30 14:32:23,677 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009)
>2014-07-30 14:32:23,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 14:32:23,683 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 14:32:23,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] FINISH, GetFileStatsVDSCommand, log id: 21a51a33
>2014-07-30 14:32:23,685 WARN  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] The refresh process from VDSM, for Floppy files failed.
>2014-07-30 14:32:23,685 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009)
>2014-07-30 14:32:23,681 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] Finished automatic refresh process for ISO file type with failure, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 14:32:23,688 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] Finished automatic refresh process for Floppy file type with failure, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71.
>2014-07-30 14:32:23,690 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Correlation ID: 2a1cf137, Job ID: 95754423-1a75-479b-9b6b-917dc1515c25, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 14:32:23,701 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Removing vds [] from the domain in maintenance cache
>2014-07-30 14:32:23,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 14:32:23,704 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [58bfe349] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 14:32:23,706 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [58bfe349] The following operation Event [eventType=DOMAINMONITORING, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=null, vdsId=17f7fe56-5f32-40ef-b098-611b84484e0b, description=] was cancelled, because of reconstruct was run before
>2014-07-30 14:32:23,738 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (DefaultQuartzScheduler_Worker-58) [7b19edd8] The following storage domains had a problem retrieving data from VDSM  ( Storage Pool Id: 00000002-0002-0002-0002-0000000001d9 Storage domain Id: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 File type: ISO)  ( Storage Pool Id: 00000002-0002-0002-0002-0000000001d9 Storage domain Id: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 File type: Floppy) 
>2014-07-30 14:32:23,743 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list failed for domain(s):   str01-iso-shared (ISO file type)  str01-iso-shared (Floppy file type). Please check domain activity.
>2014-07-30 14:32:26,806 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-12) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-30 14:32:26,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-07-30 14:32:26,820 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-30 14:32:26,827 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-30 14:32:30,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 60cf4a11
>2014-07-30 14:32:30,113 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, GetHardwareInfoVDSCommand, log id: 60cf4a11
>2014-07-30 14:32:30,146 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [767bfeb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 14:32:30,155 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [767bfeb3] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-30 14:32:30,163 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [374f1b92] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 14:32:30,171 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 14:32:30,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2c036780
>2014-07-30 14:32:30,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 2c036780
>2014-07-30 14:32:30,711 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] Host rack storage connection was succeeded 
>2014-07-30 14:32:30,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 68fe8d22
>2014-07-30 14:32:33,666 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 14:32:33,670 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 14:33:37,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, ConnectStoragePoolVDSCommand, log id: 68fe8d22
>2014-07-30 14:33:37,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 9e0610
>2014-07-30 14:33:37,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] FINISH, SetMOMPolicyParametersVDSCommand, log id: 9e0610
>2014-07-30 14:33:37,143 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45b82729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-30 14:33:37,149 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45b82729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-30 14:33:37,167 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 14:33:43,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 14:33:43,837 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 14:33:43,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52394d74
>2014-07-30 14:33:43,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling started: taskId = 20f424e1-7097-4019-952c-2de62e8f5aee
>2014-07-30 14:33:44,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended: taskId = 20f424e1-7097-4019-952c-2de62e8f5aee task status = finished
>2014-07-30 14:33:44,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended, spm status: SPM
>2014-07-30 14:33:44,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=20f424e1-7097-4019-952c-2de62e8f5aee), log id: 731228b1
>2014-07-30 14:33:44,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, HSMClearTaskVDSCommand, log id: 731228b1
>2014-07-30 14:33:44,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ecef617, log id: 52394d74
>2014-07-30 14:33:45,008 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 14:33:45,014 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 14:33:45,016 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-30 14:33:45,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39420802
>2014-07-30 14:33:46,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 14:33:46,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15a92800
>2014-07-30 14:33:46,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 15a92800
>2014-07-30 14:33:46,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39420802
>2014-07-30 14:33:47,000 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-50) Discovered no tasks on Storage Pool Default
>2014-07-30 15:08:18,836 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [20a126cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:08:18,932 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 15:08:18,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 5b362203
>2014-07-30 15:08:18,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] FINISH, DeactivateStorageDomainVDSCommand, log id: 5b362203
>2014-07-30 15:08:18,962 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:08:18,971 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-32) After deactivate treatment vds: rack,pool Default
>2014-07-30 15:08:18,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4bfd7af2
>2014-07-30 15:09:25,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 4bfd7af2
>2014-07-30 15:09:25,133 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:09:25,135 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Adding domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to the domains in maintenance cache
>2014-07-30 15:09:25,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43d446e2
>2014-07-30 15:09:25,190 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:09:25,197 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:09:25,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:09:25,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:09:25,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:09:25,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] FINISH, DisconnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 43d446e2
>2014-07-30 15:09:25,364 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 will remain in PreparingForMaintenance status until deactivated on all hosts
>2014-07-30 15:09:25,376 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Correlation ID: 20a126cf, Job ID: 00122294-d65c-427d-bd65-7ec499615540, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by admin
>2014-07-30 15:09:35,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance
>2014-07-30 15:18:59,554 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 15:18:59,558 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Start detach storage domain
>2014-07-30 15:18:59,563 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15]  Detach storage domain: before connect
>2014-07-30 15:18:59,567 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:18:59,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2ecb99fb
>2014-07-30 15:18:59,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 2ecb99fb
>2014-07-30 15:18:59,647 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15]  Detach storage domain: after connect
>2014-07-30 15:18:59,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 6eaeea5f
>2014-07-30 15:18:59,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 from the domains in maintenance cache
>2014-07-30 15:19:02,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] FINISH, DetachStorageDomainVDSCommand, log id: 6eaeea5f
>2014-07-30 15:19:02,946 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15]  Detach storage domain: after detach in vds
>2014-07-30 15:19:02,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 51703187
>2014-07-30 15:19:18,055 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:19:18,063 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Running command: RemoveStorageDomainCommand internal: false. Entities affected :  ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage
>2014-07-30 15:19:18,160 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [39bbddf7] Correlation ID: 39bbddf7, Job ID: aa86eacb-39d8-4506-87a8-366ba61bc633, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was removed by admin
>2014-07-30 15:19:18,167 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:20:05,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 51703187
>2014-07-30 15:20:05,069 WARN  [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-18) Did not connect host: 17f7fe56-5f32-40ef-b098-611b84484e0b to storage domain: str01-iso-shared because connection for connectionId:51e002a3-3547-4fbf-b3c9-6a4536cca8a8 is null.
>2014-07-30 15:20:05,071 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15]  Detach storage domain: after disconnect storage
>2014-07-30 15:20:05,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] START, ResetISOPathVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39d1c3a1
>2014-07-30 15:20:05,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] FINISH, ResetISOPathVDSCommand, log id: 39d1c3a1
>2014-07-30 15:20:05,170 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] End detach storage domain
>2014-07-30 15:20:05,178 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [10039d15] Correlation ID: 10039d15, Job ID: 6d926e1a-877e-4402-9f30-a71ac1f9db6c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was detached from Data Center Default by admin
>2014-07-30 15:23:13,482 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-07-30 15:23:13,492 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:23:13,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6563eb38
>2014-07-30 15:23:55,145 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) [296dca52] Principal admin::internal synchronized
>2014-07-30 15:24:22,086 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf.
>2014-07-30 15:25:23,566 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [1286263f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN.
>2014-07-30 15:25:23,567 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 6563eb38
>2014-07-30 15:25:23,570 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [1286263f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target
>2014-07-30 15:25:23,572 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-50) [1286263f] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target
>2014-07-30 15:25:23,578 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-30 15:25:23,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: b91988e
>2014-07-30 15:27:24,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Failed in AttachStorageDomainVDS method
>2014-07-30 15:27:24,146 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358
>2014-07-30 15:27:24,156 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] FINISH, AttachStorageDomainVDSCommand, log id: b91988e
>2014-07-30 15:27:24,158 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
>2014-07-30 15:27:24,163 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Command [id=9295ce3e-dde8-4d91-ac4e-368395f4d41d]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-07-30 15:27:24,175 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [63580ade] Correlation ID: 63580ade, Job ID: eed628df-27b6-4c89-9482-c76ff2b562c0, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin)
>2014-07-30 15:33:59,448 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-07-30 15:33:59,460 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:33:59,464 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 747b3d4e
>2014-07-30 15:36:09,528 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN.
>2014-07-30 15:36:09,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 747b3d4e
>2014-07-30 15:36:09,532 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target
>2014-07-30 15:36:09,534 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-20) [26e7de54] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target
>2014-07-30 15:36:09,538 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-30 15:36:09,540 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 26013a9b
>2014-07-30 15:38:09,967 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Failed in AttachStorageDomainVDS method
>2014-07-30 15:38:10,000 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358
>2014-07-30 15:38:10,010 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] FINISH, AttachStorageDomainVDSCommand, log id: 26013a9b
>2014-07-30 15:38:10,011 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
>2014-07-30 15:38:10,017 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Command [id=7d68ad4e-6f96-4ed6-8fff-9a5456a4cad2]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-07-30 15:38:10,029 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [4b747578] Correlation ID: 4b747578, Job ID: d2fd16d1-43cc-4ff2-9e72-cae105b93986, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin)
>2014-07-30 15:43:24,401 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 15:43:24,412 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:43:24,415 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-3) [40864998] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bdcd55
>2014-07-30 15:43:25,107 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-3) [40864998] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4bdcd55
>2014-07-30 15:43:25,115 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 15:43:25,220 INFO  [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-11) [43c29a81] Running command: AddNFSStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:43:25,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75251f19
>2014-07-30 15:43:25,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 75251f19
>2014-07-30 15:43:25,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@a1a50139, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export), log id: 6b3a6865
>2014-07-30 15:44:26,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, CreateStorageDomainVDSCommand, log id: 6b3a6865
>2014-07-30 15:44:26,215 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 70991c2c
>2014-07-30 15:44:26,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@5eab6273, log id: 70991c2c
>2014-07-30 15:44:26,279 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [43c29a81] Correlation ID: 43c29a81, Job ID: 497a0273-48d0-4630-bf2e-6b68a26ffa45, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was added by admin
>2014-07-30 15:44:26,679 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:44:26,682 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 716ea070
>2014-07-30 15:44:26,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 716ea070
>2014-07-30 15:44:27,347 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage
>2014-07-30 15:44:27,357 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:44:27,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5254a13d
>2014-07-30 15:44:27,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 5254a13d
>2014-07-30 15:44:27,407 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 49cce2a3
>2014-07-30 15:44:27,630 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@b31ea5b3, null>, log id: 49cce2a3
>2014-07-30 15:44:27,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 2adbe1a2
>2014-07-30 15:44:51,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, AttachStorageDomainVDSCommand, log id: 2adbe1a2
>2014-07-30 15:44:51,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 594c9668
>2014-07-30 15:44:51,887 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, GetImagesListVDSCommand, return: [], log id: 594c9668
>2014-07-30 15:44:51,894 WARN  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-6) There are no OVF_STORE disks on storage domain id 453fe8a4-ef9b-4952-bf43-756dc497291e
>2014-07-30 15:44:51,900 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:44:51,911 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage
>2014-07-30 15:44:51,916 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= ]
>2014-07-30 15:44:51,917 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 3:44 PM
>2014-07-30 15:44:51,921 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 15:44:51,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e8de28a
>2014-07-30 15:44:51,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 5e8de28a
>2014-07-30 15:44:51,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [112350cc] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 47005d16
>2014-07-30 15:44:52,174 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [112350cc] FINISH, ActivateStorageDomainVDSCommand, log id: 47005d16
>2014-07-30 15:44:52,175 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 3:44 PM
>2014-07-30 15:44:52,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bc61ac
>2014-07-30 15:45:52,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, ConnectStoragePoolVDSCommand, log id: 2bc61ac
>2014-07-30 15:45:52,430 INFO  [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-8) Refreshed vds rack in pool Default
>2014-07-30 15:45:52,431 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 3:45 PM
>2014-07-30 15:45:52,437 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [112350cc] Correlation ID: 112350cc, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by Non interactive user
>2014-07-30 15:45:52,443 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [112350cc] Correlation ID: 7c810b8b, Job ID: 98fe282b-4697-43b5-89a9-b3a389815624, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was attached to Data Center Default by admin
>2014-07-30 16:01:49,627 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null
>2014-07-30 16:02:06,951 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-07-30 16:02:06,958 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 16:02:06,975 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:02:07,063 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:02:07,841 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:02:29,370 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [1d31befe] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-30 16:02:29,393 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage
>2014-07-30 16:02:29,400 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 17d57ccd
>2014-07-30 16:02:32,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] FINISH, DeactivateStorageDomainVDSCommand, log id: 17d57ccd
>2014-07-30 16:02:32,416 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-07-30 16:02:32,426 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-46) After deactivate treatment vds: rack,pool Default
>2014-07-30 16:02:32,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 254c628d
>2014-07-30 16:03:32,731 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 254c628d
>2014-07-30 16:03:32,739 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:03:32,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 16e0a177
>2014-07-30 16:03:32,795 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Adding domain 453fe8a4-ef9b-4952-bf43-756dc497291e to the domains in maintenance cache
>2014-07-30 16:03:32,799 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance
>2014-07-30 16:03:32,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance
>2014-07-30 16:03:32,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance
>2014-07-30 16:03:32,816 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance
>2014-07-30 16:03:32,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 16e0a177
>2014-07-30 16:03:32,947 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Domain 453fe8a4-ef9b-4952-bf43-756dc497291e will remain in PreparingForMaintenance status until deactivated on all hosts
>2014-07-30 16:03:32,956 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Correlation ID: 1d31befe, Job ID: 251985e0-17b4-4d35-ad28-6754cc9c1449, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin
>2014-07-30 16:03:34,069 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance
>2014-07-30 16:05:05,843 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected :  ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage
>2014-07-30 16:05:05,845 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Start detach storage domain
>2014-07-30 16:05:05,849 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6]  Detach storage domain: before connect
>2014-07-30 16:05:05,854 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:05:05,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 33c2e97
>2014-07-30 16:05:05,916 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 33c2e97
>2014-07-30 16:05:05,919 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6]  Detach storage domain: after connect
>2014-07-30 16:05:05,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 769677f2
>2014-07-30 16:05:06,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] FINISH, DetachStorageDomainVDSCommand, log id: 769677f2
>2014-07-30 16:05:06,108 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6]  Detach storage domain: after detach in vds
>2014-07-30 16:05:06,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5a072551
>2014-07-30 16:06:06,419 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5a072551
>2014-07-30 16:06:06,422 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:06:06,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2954ffef
>2014-07-30 16:06:06,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 2954ffef
>2014-07-30 16:06:06,622 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6]  Detach storage domain: after disconnect storage
>2014-07-30 16:06:06,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Removing domain 453fe8a4-ef9b-4952-bf43-756dc497291e from the domains in maintenance cache
>2014-07-30 16:06:06,635 INFO  [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] End detach storage domain
>2014-07-30 16:06:06,641 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Correlation ID: 675e19b6, Job ID: 227d28da-8129-41c9-873f-f1451ed7d3eb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was detached from Data Center Default by admin
>2014-07-30 16:06:09,278 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [226334ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:06:09,288 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [226334ea] Running command: RemoveStorageDomainCommand internal: false. Entities affected :  ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage
>2014-07-30 16:06:09,291 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-2) [674b6284] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:06:09,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7a008e50
>2014-07-30 16:06:09,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 7a008e50
>2014-07-30 16:06:09,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] START, FormatStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 55dc326c
>2014-07-30 16:06:09,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] FINISH, FormatStorageDomainVDSCommand, log id: 55dc326c
>2014-07-30 16:06:09,809 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [10e97f08] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:06:09,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [10e97f08] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 581e493
>2014-07-30 16:06:09,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [10e97f08] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 581e493
>2014-07-30 16:06:10,068 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [10e97f08] Correlation ID: 226334ea, Job ID: f1e8c884-9ab1-44e7-8c72-d30fcab21078, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was removed by admin
>2014-07-30 16:06:10,074 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [10e97f08] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:06:56,566 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 16:06:56,580 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:06:56,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [387f6226] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4320772c
>2014-07-30 16:06:56,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [387f6226] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4320772c
>2014-07-30 16:06:56,631 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-07-30 16:06:56,754 INFO  [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] Running command: AddNFSStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:06:56,766 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28834bf6
>2014-07-30 16:06:56,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 28834bf6
>2014-07-30 16:06:56,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@3a6e99d4, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export), log id: 6623ab32
>2014-07-30 16:07:57,529 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, CreateStorageDomainVDSCommand, log id: 6623ab32
>2014-07-30 16:07:57,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=44559ec0-6bef-491c-bbd6-e67278f37987), log id: 19da4130
>2014-07-30 16:07:57,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@9c054809, log id: 19da4130
>2014-07-30 16:07:57,599 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [24d39fbe] Correlation ID: 24d39fbe, Job ID: 151e336d-46e4-426a-b84e-73c9286efb3a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was added by admin
>2014-07-30 16:07:57,714 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [4da90475] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:07:57,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [4da90475] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 23079a1a
>2014-07-30 16:07:57,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [4da90475] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 23079a1a
>2014-07-30 16:07:58,110 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-11) [62268b88] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 16:07:58,121 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:07:58,127 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5a304423
>2014-07-30 16:07:58,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5a304423
>2014-07-30 16:07:58,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 214d7abf
>2014-07-30 16:08:19,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] FINISH, AttachStorageDomainVDSCommand, log id: 214d7abf
>2014-07-30 16:08:19,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 26400e66
>2014-07-30 16:08:20,050 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] FINISH, GetImagesListVDSCommand, return: [], log id: 26400e66
>2014-07-30 16:08:20,057 WARN  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-11) There are no OVF_STORE disks on storage domain id 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-07-30 16:08:20,065 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:08:20,079 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 16:08:20,086 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:08:20,087 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:08 PM
>2014-07-30 16:08:20,092 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:08:20,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e311b55
>2014-07-30 16:08:20,110 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e311b55
>2014-07-30 16:08:20,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [3711efe3] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 17bc6586
>2014-07-30 16:08:20,349 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [3711efe3] FINISH, ActivateStorageDomainVDSCommand, log id: 17bc6586
>2014-07-30 16:08:20,350 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 4:08 PM
>2014-07-30 16:08:20,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: c855534
>2014-07-30 16:09:04,456 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-07-30 16:09:04,464 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 16:09:04,465 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:09:04,552 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:09:04,580 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:09:20,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, ConnectStoragePoolVDSCommand, log id: c855534
>2014-07-30 16:09:20,658 INFO  [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-8) Refreshed vds rack in pool Default
>2014-07-30 16:09:20,670 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 4:09 PM
>2014-07-30 16:09:20,677 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [3711efe3] Correlation ID: 3711efe3, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by Non interactive user
>2014-07-30 16:09:20,688 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [3711efe3] Correlation ID: 62268b88, Job ID: 83c9e166-e27f-42ff-a514-5cc368113dc2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was attached to Data Center Default by admin
>2014-07-30 16:09:43,243 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-07-30 16:09:43,251 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 16:09:43,254 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:09:43,352 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:11:17,551 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-07-30 16:11:17,555 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 16:11:17,556 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:11:17,727 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:13:48,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 5994058d
>2014-07-30 16:13:48,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, GetVmsInfoVDSCommand, log id: 5994058d
>2014-07-30 16:13:55,817 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-1) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 4470d32e
>2014-07-30 16:13:55,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, GetVmsInfoVDSCommand, log id: 4470d32e
>2014-07-30 16:14:12,653 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-3) [cac56be] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:14:12,676 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 16:14:12,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 299791ba
>2014-07-30 16:14:12,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] FINISH, DeactivateStorageDomainVDSCommand, log id: 299791ba
>2014-07-30 16:14:12,698 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:14:12,708 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-32) After deactivate treatment vds: rack,pool Default
>2014-07-30 16:14:12,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 752d8055
>2014-07-30 16:15:12,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 752d8055
>2014-07-30 16:15:12,965 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:15:12,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache
>2014-07-30 16:15:12,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:15:12,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4aba71d2
>2014-07-30 16:15:12,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:15:12,986 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:15:12,991 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:15:12,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:15:13,166 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4aba71d2
>2014-07-30 16:15:13,171 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] Domain 44559ec0-6bef-491c-bbd6-e67278f37987 will remain in PreparingForMaintenance status until deactivated on all hosts
>2014-07-30 16:15:13,181 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [52520451] Correlation ID: cac56be, Job ID: 28d66c42-7ba2-42ff-afd7-eb605a369439, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin
>2014-07-30 16:15:22,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-07-30 16:17:13,328 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [6df02cad] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:17:13,361 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 16:17:13,367 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:17:13,369 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:17 PM
>2014-07-30 16:17:13,374 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:17:13,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7af25f08
>2014-07-30 16:17:13,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7af25f08
>2014-07-30 16:17:13,439 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 7c841d9a
>2014-07-30 16:17:13,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] FINISH, ActivateStorageDomainVDSCommand, log id: 7c841d9a
>2014-07-30 16:17:13,677 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 4:17 PM
>2014-07-30 16:17:13,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 55f43dd4
>2014-07-30 16:18:13,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 55f43dd4
>2014-07-30 16:18:13,951 INFO  [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-12) Refreshed vds rack in pool Default
>2014-07-30 16:18:13,952 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 4:18 PM
>2014-07-30 16:18:13,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Removing domain 44559ec0-6bef-491c-bbd6-e67278f37987 from the domains in maintenance cache
>2014-07-30 16:18:13,961 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Correlation ID: 6df02cad, Job ID: d71d1d44-dea9-431e-af30-11724b0f4324, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by admin
>2014-07-30 16:18:29,070 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 16:18:29,072 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-30 16:18:29,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Host 17f7fe56-5f32-40ef-b098-611b84484e0b has reported new storage access problem to the following domains 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752,44559ec0-6bef-491c-bbd6-e67278f37987 marking it for storage connections and pool metadata refresh (report id: e8a103cd-8125-46cd-bd80-a65117c56e2f)
>2014-07-30 16:18:29,302 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Running storage connections refresh for hosts [17f7fe56-5f32-40ef-b098-611b84484e0b]
>2014-07-30 16:18:29,305 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 16:18:29,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5cc65080
>2014-07-30 16:20:29,332 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5cc65080
>2014-07-30 16:20:29,333 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] Host rack storage connection was succeeded 
>2014-07-30 16:20:29,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Submitting to the event queue pool refresh for hosts [17f7fe56-5f32-40ef-b098-611b84484e0b]
>2014-07-30 16:20:29,336 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Running storage pool metadata refresh for hosts {1}
>2014-07-30 16:20:29,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 33b2f639
>2014-07-30 16:21:35,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 16:21:35,770 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 16:21:35,774 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-30 16:21:35,778 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 16:21:35,795 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [6bd9692] Correlation ID: 6bd9692, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 16:21:55,083 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-6) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 2ae6fcc0
>2014-07-30 16:22:35,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value 
> StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-0000000001d9, msdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752']]
>2014-07-30 16:22:35,811 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) HostName = rack
>2014-07-30 16:22:35,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-0000000001d9, msdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752'
>2014-07-30 16:22:35,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 33b2f639
>2014-07-30 16:23:29,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 16:23:29,080 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 16:23:29,119 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 16:23:29,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 52b4dda9
>2014-07-30 16:23:29,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1ceee97e
>2014-07-30 16:23:29,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 16:23:32,340 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [72233a31] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:23:32,362 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 16:23:32,367 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:23:32,368 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:23 PM
>2014-07-30 16:23:32,373 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-30 16:23:32,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 25be2568
>2014-07-30 16:23:55,153 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Principal admin::internal synchronized
>2014-07-30 16:24:22,116 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf.
>2014-07-30 16:24:35,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 16:24:35,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) [6bd9692] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 16:24:35,833 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-30 16:24:35,834 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 16:25:35,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 16:25:35,785 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-30 16:25:35,797 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] FINISH, SpmStopVDSCommand, log id: 1ceee97e
>2014-07-30 16:25:35,798 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] FINISH, ConnectStorageServerVDSCommand, log id: 25be2568
>2014-07-30 16:25:35,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (ajp--127.0.0.1-8702-6) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 16:25:35,803 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-6) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 16:25:35,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-71) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 16:25:35,804 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-30 16:25:35,806 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-30 16:25:35,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] FINISH, ResetIrsVDSCommand, log id: 52b4dda9
>2014-07-30 16:25:35,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetVmsInfoVDSCommand, log id: 2ae6fcc0
>2014-07-30 16:25:35,821 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-30 16:25:35,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 16:25:35,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) [5aa74089] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 16:25:35,830 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to retrieve VM/Templates information from export domain export
>2014-07-30 16:25:35,827 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Correlation ID: 266abdd1, Job ID: ad361002-0602-4453-adab-899c1a473485, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 16:25:35,827 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-30 16:25:35,860 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 2050ebd4
>2014-07-30 16:25:35,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Removing vds [] from the domain in maintenance cache
>2014-07-30 16:25:35,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] FINISH, ActivateStorageDomainVDSCommand, log id: 2050ebd4
>2014-07-30 16:25:35,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 16:25:35,868 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [375fcc6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 16:25:35,867 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009)
>2014-07-30 16:25:35,855 ERROR [org.ovirt.engine.core.bll.storage.GetVmsFromExportDomainQuery] (ajp--127.0.0.1-8702-6) Query GetVmsFromExportDomainQuery failed. Exception message is VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) : org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009)
>	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:]
>	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
>	at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.executeVerb(GetAllFromExportDomainQuery.java:60) [bll.jar:]
>	at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.getAllFromStorage(GetAllFromExportDomainQuery.java:48) [bll.jar:]
>	at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.executeQueryCommand(GetAllFromExportDomainQuery.java:37) [bll.jar:]
>	at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:73) [bll.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:492) [bll.jar:]
>	at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:466) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view7.runQuery(Unknown Source) [common.jar:]
>	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runQuery(GenericApiGWTServiceImpl.java:56)
>	at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196)
>	at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172)
>	at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233)
>	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
>	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
>	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72)
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.SessionMgmtFilter.doFilter(SessionMgmtFilter.java:31) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
>	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
>	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
>	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
>	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
>	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
>	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
>	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
>	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
>	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-30 16:25:35,867 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-30 16:25:35,880 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Command [id=00b238d2-2e70-4a3c-b47b-1c976d071d2c]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, status=Inactive].
>2014-07-30 16:25:35,877 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [375fcc6] The following operation Event [eventType=RECONSTRUCT, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, vdsId=null, description=IrsBrokerCommand.startReconstruct()] was cancelled, because of reconstruct was run before
>2014-07-30 16:25:35,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-30 16:25:35,950 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-30 16:25:35,955 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [72233a31] Correlation ID: 72233a31, Job ID: 79281f4f-e62b-42ad-8e65-a5ec7b89f594, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain 35 (Data Center Default) by admin
>2014-07-30 16:25:35,973 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:25:35,981 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 16:25:35,988 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 16:25:36,012 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Correlation ID: b2cf278, Job ID: fa4aadbe-e5a7-4068-99ae-8a43e205f9ec, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-30 16:25:36,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Removing vds [] from the domain in maintenance cache
>2014-07-30 16:25:36,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 16:25:39,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: a95296c
>2014-07-30 16:25:39,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GetHardwareInfoVDSCommand, log id: a95296c
>2014-07-30 16:25:39,281 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [2f820231] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 16:25:39,294 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-97) [57113c30] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 16:25:39,303 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 16:25:39,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ee2805a
>2014-07-30 16:25:39,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5ee2805a
>2014-07-30 16:25:39,379 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] Host rack storage connection was succeeded 
>2014-07-30 16:25:39,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2b99de1
>2014-07-30 16:25:45,857 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 16:25:45,861 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 16:26:39,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 2b99de1
>2014-07-30 16:26:39,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e2d6a98
>2014-07-30 16:26:39,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4e2d6a98
>2014-07-30 16:26:39,897 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [5971df3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-30 16:26:39,903 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [5971df3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-30 16:26:39,925 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 16:26:39,934 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Host 17f7fe56-5f32-40ef-b098-611b84484e0b no longer storage access problem to any relevant domain  clearing its report (report id: e8a103cd-8125-46cd-bd80-a65117c56e2f)
>2014-07-30 16:26:45,828 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) No hosts has reported storage access problem to domains, clearing the handled hosts reports map
>2014-07-30 16:26:46,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 16:26:46,018 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 16:26:46,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78d02718
>2014-07-30 16:26:46,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling started: taskId = bff5113e-5416-493f-b37f-486fb4a4e9ed
>2014-07-30 16:26:47,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling ended: taskId = bff5113e-5416-493f-b37f-486fb4a4e9ed task status = finished
>2014-07-30 16:26:47,132 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling ended, spm status: SPM
>2014-07-30 16:26:47,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=bff5113e-5416-493f-b37f-486fb4a4e9ed), log id: 3cb14a3e
>2014-07-30 16:26:47,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, HSMClearTaskVDSCommand, log id: 3cb14a3e
>2014-07-30 16:26:47,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a3d4447, log id: 78d02718
>2014-07-30 16:26:47,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 16:26:47,193 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 16:26:47,207 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10637abb
>2014-07-30 16:26:47,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 16:26:47,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2b7607c7
>2014-07-30 16:26:47,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b7607c7
>2014-07-30 16:26:47,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 10637abb
>2014-07-30 16:26:47,248 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] Discovered no tasks on Storage Pool Default
>2014-07-30 16:39:51,917 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-07-30 16:39:51,923 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-30 16:39:51,924 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 16:39:52,020 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-07-30 17:14:57,940 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-11) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 5b9d755d
>2014-07-30 17:14:57,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-11) FINISH, GetVmsInfoVDSCommand, log id: 5b9d755d
>2014-07-30 17:15:00,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 15e5ed43
>2014-07-30 17:15:00,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, GetVmsInfoVDSCommand, log id: 15e5ed43
>2014-07-30 17:23:55,160 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Principal admin::internal synchronized
>2014-07-30 17:24:22,119 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf.
>2014-07-30 18:23:55,166 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-07-30 18:24:22,151 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf.
>2014-07-30 19:23:55,172 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized
>2014-07-30 19:24:22,168 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf.
>2014-07-30 19:26:36,490 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 19:26:36,493 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-30 19:28:03,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 19:28:03,171 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 19:28:03,175 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-8) [782c75f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-30 19:28:03,177 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-8) [782c75f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 19:28:03,190 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: 782c75f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 19:30:03,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 19:30:03,224 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 19:30:03,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-30 19:30:03,226 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 19:30:03,261 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 19:30:03,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 252f9f2c
>2014-07-30 19:30:03,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 68705a37
>2014-07-30 19:30:03,291 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 19:32:03,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 19:32:03,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 19:32:03,199 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-30 19:32:03,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] FINISH, SpmStopVDSCommand, log id: 68705a37
>2014-07-30 19:32:03,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-12) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 19:32:03,206 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-30 19:32:03,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] FINISH, ResetIrsVDSCommand, log id: 252f9f2c
>2014-07-30 19:32:03,208 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-30 19:32:03,218 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] Correlation ID: 6a94c2dc, Job ID: 2573d5b8-e851-427a-a9d4-6f043c93cfce, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 19:32:03,241 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [782c75f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 19:32:03,241 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-30 19:32:03,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 19:32:03,300 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 19:32:03,320 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [16ecab23] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-30 19:32:03,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [16ecab23] Removing vds [] from the domain in maintenance cache
>2014-07-30 19:32:03,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [16ecab23] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 19:32:03,329 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [782c75f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 19:32:03,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-30 19:32:03,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-30 19:32:03,362 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 19:32:03,368 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 19:32:03,374 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 19:32:03,382 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [18825565] Correlation ID: 18825565, Job ID: f76c3eb9-55ab-48b6-b0ac-414a5345fafe, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-30 19:32:03,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [18825565] Removing vds [] from the domain in maintenance cache
>2014-07-30 19:32:03,388 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [18825565] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 19:32:06,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 54d96ce6
>2014-07-30 19:32:06,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, GetHardwareInfoVDSCommand, log id: 54d96ce6
>2014-07-30 19:32:06,608 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [3514063c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 19:32:06,624 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [12346060] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 19:32:06,638 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 19:32:06,658 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f61d7ad
>2014-07-30 19:32:06,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: f61d7ad
>2014-07-30 19:32:06,677 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] Host rack storage connection was succeeded 
>2014-07-30 19:32:06,684 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4e27dc37
>2014-07-30 19:32:13,203 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-73) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 19:32:13,207 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-73) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 19:33:07,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 4e27dc37
>2014-07-30 19:33:07,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 184049e8
>2014-07-30 19:33:07,168 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] FINISH, SetMOMPolicyParametersVDSCommand, log id: 184049e8
>2014-07-30 19:33:07,174 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [756c1100] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-30 19:33:07,180 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [756c1100] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-30 19:33:07,202 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 19:33:13,318 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 19:33:13,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 19:33:13,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 9ae6754
>2014-07-30 19:33:13,373 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling started: taskId = 592047e7-270a-46f3-b11d-357579b2bea4
>2014-07-30 19:33:14,385 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling ended: taskId = 592047e7-270a-46f3-b11d-357579b2bea4 task status = finished
>2014-07-30 19:33:14,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling ended, spm status: SPM
>2014-07-30 19:33:14,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=592047e7-270a-46f3-b11d-357579b2bea4), log id: 3254ecf3
>2014-07-30 19:33:14,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, HSMClearTaskVDSCommand, log id: 3254ecf3
>2014-07-30 19:33:14,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69eef01b, log id: 9ae6754
>2014-07-30 19:33:14,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 19:33:14,499 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 19:33:14,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: c61f44a
>2014-07-30 19:33:14,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 19:33:14,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 50ad254f
>2014-07-30 19:33:14,558 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50ad254f
>2014-07-30 19:33:14,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c61f44a
>2014-07-30 19:33:14,561 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] Discovered no tasks on Storage Pool Default
>2014-07-30 20:23:55,181 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized
>2014-07-30 20:24:22,183 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
>2014-07-30 21:23:55,189 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized
>2014-07-30 21:24:22,213 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-07-30 22:23:55,195 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized
>2014-07-30 22:24:22,248 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
>2014-07-30 22:32:54,700 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-30 22:32:54,702 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-30 22:34:29,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 22:34:29,007 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 22:34:29,010 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-30 22:34:29,012 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-30 22:34:29,025 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Correlation ID: 1f7257b7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-30 22:36:29,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 22:36:29,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-59) [1f7257b7] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 22:36:29,058 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-30 22:36:29,059 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-30 22:36:29,102 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-30 22:36:29,124 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1c8da596
>2014-07-30 22:36:29,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6f26c4a6
>2014-07-30 22:36:29,134 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-30 22:38:29,028 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-30 22:38:29,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 22:38:29,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] FINISH, SpmStopVDSCommand, log id: 6f26c4a6
>2014-07-30 22:38:29,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-30 22:38:29,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) [6bd9692] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-30 22:38:29,040 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-30 22:38:29,043 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-30 22:38:29,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-30 22:38:29,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] FINISH, ResetIrsVDSCommand, log id: 1c8da596
>2014-07-30 22:38:29,091 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-30 22:38:29,094 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] Correlation ID: 4eb6f81d, Job ID: 2ecec18f-a4b2-4caa-84c8-ac9cbd7dc151, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-30 22:38:29,104 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [1f7257b7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 22:38:29,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-30 22:38:29,178 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-30 22:38:29,199 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 22:38:29,207 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-30 22:38:29,213 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-30 22:38:29,222 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [3e307027] Correlation ID: 3e307027, Job ID: b247f619-daf3-4b7b-893a-9aa235f98fa9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-30 22:38:29,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [3e307027] Removing vds [] from the domain in maintenance cache
>2014-07-30 22:38:29,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [3e307027] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 22:38:29,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-30 22:38:29,234 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-30 22:38:29,254 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [47b016c8] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-30 22:38:29,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [47b016c8] Removing vds [] from the domain in maintenance cache
>2014-07-30 22:38:29,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [47b016c8] Removing host(s) [] from hosts unseen domain report cache
>2014-07-30 22:38:29,262 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [1f7257b7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-30 22:38:32,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 35aa52cc
>2014-07-30 22:38:32,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, GetHardwareInfoVDSCommand, log id: 35aa52cc
>2014-07-30 22:38:32,435 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [1aa24b07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 22:38:32,486 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [6a81647c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 22:38:32,494 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [9604479] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-30 22:38:32,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64dac9e5
>2014-07-30 22:38:32,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 64dac9e5
>2014-07-30 22:38:32,573 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [9604479] Host rack storage connection was succeeded 
>2014-07-30 22:38:32,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4adb6926
>2014-07-30 22:38:39,046 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 22:38:39,051 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-30 22:39:32,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4adb6926
>2014-07-30 22:39:32,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 38598c78
>2014-07-30 22:39:33,089 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] FINISH, SetMOMPolicyParametersVDSCommand, log id: 38598c78
>2014-07-30 22:39:33,098 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [9604479] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-30 22:39:33,104 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [9604479] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-30 22:39:33,120 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-30 22:39:39,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-30 22:39:39,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-30 22:39:39,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b575257
>2014-07-30 22:39:39,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = 694f6db5-2ed4-4757-86df-bcd06c31407d
>2014-07-30 22:39:40,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = 694f6db5-2ed4-4757-86df-bcd06c31407d task status = finished
>2014-07-30 22:39:40,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM
>2014-07-30 22:39:40,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=694f6db5-2ed4-4757-86df-bcd06c31407d), log id: 41465994
>2014-07-30 22:39:40,382 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 41465994
>2014-07-30 22:39:40,383 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d51d68d, log id: 3b575257
>2014-07-30 22:39:40,387 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-30 22:39:40,393 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-30 22:39:40,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5888787f
>2014-07-30 22:39:40,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-30 22:39:40,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4b80b251
>2014-07-30 22:39:40,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b80b251
>2014-07-30 22:39:40,445 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5888787f
>2014-07-30 22:39:40,446 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default
>2014-07-30 23:23:55,201 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized
>2014-07-30 23:24:22,262 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-07-31 00:23:55,206 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized
>2014-07-31 00:24:22,293 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [21e62018] Attempting to update VMs/Templates Ovf.
>2014-07-31 01:23:55,214 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized
>2014-07-31 01:24:22,311 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf.
>2014-07-31 01:39:26,958 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 01:39:26,959 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 01:40:56,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 01:40:56,865 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 01:40:56,869 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [11b7484a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 01:40:56,872 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [11b7484a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 01:40:56,888 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [11b7484a] Correlation ID: 11b7484a, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 01:42:56,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 01:42:56,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [11b7484a] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 01:42:56,931 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 01:42:56,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 01:42:56,969 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 01:42:56,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 65247c5e
>2014-07-31 01:42:56,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 36aa66fb
>2014-07-31 01:42:56,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 01:44:56,886 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 01:44:56,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] FINISH, SpmStopVDSCommand, log id: 36aa66fb
>2014-07-31 01:44:56,894 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 01:44:56,896 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 01:44:56,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 01:44:56,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] FINISH, ResetIrsVDSCommand, log id: 65247c5e
>2014-07-31 01:44:56,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 01:44:56,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 01:44:56,939 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 01:44:56,943 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [251e5084] Correlation ID: 251e5084, Job ID: 181abcb7-3270-4342-b5c3-bdd8db678191, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 01:44:56,948 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [11b7484a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 01:44:56,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 01:44:57,010 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 01:44:57,027 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 01:44:57,034 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 01:44:57,039 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 01:44:57,048 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [3b951098] Correlation ID: 3b951098, Job ID: b5b3da83-baef-485a-be12-654af7baba83, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 01:44:57,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [3b951098] Removing vds [] from the domain in maintenance cache
>2014-07-31 01:44:57,053 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [3b951098] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 01:44:57,055 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 01:44:57,060 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 01:44:57,078 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 01:44:57,084 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] Removing vds [] from the domain in maintenance cache
>2014-07-31 01:44:57,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 01:44:57,086 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [11b7484a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 01:45:00,014 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering 2 storage domains
>2014-07-31 01:45:00,015 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-31 01:45:00,017 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 01:45:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 1:45 AM
>2014-07-31 01:45:00,026 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 1:45 AM
>2014-07-31 01:45:00,029 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [68befb52] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-07-31 01:45:00,034 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 01:45:00,035 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 1:45 AM
>2014-07-31 01:45:00,042 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 1:45 AM
>2014-07-31 01:45:00,045 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e9ec75f
>2014-07-31 01:45:00,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = rack
>2014-07-31 01:45:00,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-31 01:45:00,064 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-31 01:45:00,071 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-31 01:45:03,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 69d71ab6
>2014-07-31 01:45:03,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] FINISH, GetHardwareInfoVDSCommand, log id: 69d71ab6
>2014-07-31 01:45:03,399 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [2dbcdcde] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 01:45:03,411 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) [2dbcdcde] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-31 01:45:03,418 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [4267262c] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 01:45:03,433 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 01:45:03,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f3531e0
>2014-07-31 01:45:03,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4f3531e0
>2014-07-31 01:45:03,468 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Host rack storage connection was succeeded 
>2014-07-31 01:45:03,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1fcccaa2
>2014-07-31 01:45:06,936 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 01:45:06,940 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 01:46:03,880 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 1fcccaa2
>2014-07-31 01:46:03,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f2e4816
>2014-07-31 01:46:03,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7f2e4816
>2014-07-31 01:46:03,998 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 01:46:04,005 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 01:46:04,037 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 01:46:07,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 01:46:07,119 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 01:46:07,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d2768
>2014-07-31 01:46:07,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling started: taskId = 19ff676c-b136-4ec9-8767-8b277c26fcaf
>2014-07-31 01:46:08,176 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling ended: taskId = 19ff676c-b136-4ec9-8767-8b277c26fcaf task status = finished
>2014-07-31 01:46:08,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling ended, spm status: SPM
>2014-07-31 01:46:08,235 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=19ff676c-b136-4ec9-8767-8b277c26fcaf), log id: 3bf7656a
>2014-07-31 01:46:08,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, HSMClearTaskVDSCommand, log id: 3bf7656a
>2014-07-31 01:46:08,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e5cfe61, log id: 32d2768
>2014-07-31 01:46:08,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 01:46:08,293 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 01:46:08,295 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 01:46:08,323 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7fa108de
>2014-07-31 01:46:08,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 01:46:08,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 300a2729
>2014-07-31 01:46:08,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 300a2729
>2014-07-31 01:46:08,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fa108de
>2014-07-31 01:46:08,358 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [52d379dc] Discovered no tasks on Storage Pool Default
>2014-07-31 02:23:55,225 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized
>2014-07-31 02:24:22,344 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-07-31 03:23:55,234 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized
>2014-07-31 03:24:22,358 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf.
>2014-07-31 03:35:35,000 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Start deleteAgedOutCommandEntities
>2014-07-31 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Start deleteAgedOutAuditLogs
>2014-07-31 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Finished deleteAgedOutCommandEntities
>2014-07-31 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Finished deleteAgedOutAuditLogs
>2014-07-31 04:23:55,241 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized
>2014-07-31 04:24:22,392 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf.
>2014-07-31 04:45:45,100 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 04:46:00,213 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 04:47:26,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 04:47:26,865 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 04:47:26,869 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-13) [519c108f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 04:47:26,871 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-13) [519c108f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 04:47:26,888 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [519c108f] Correlation ID: 519c108f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 04:49:26,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 04:49:26,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 04:49:26,923 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 04:49:26,924 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 04:49:26,969 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 04:49:26,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1f761dbc
>2014-07-31 04:49:26,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f1fb654
>2014-07-31 04:49:27,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 04:50:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) Autorecovering 1 storage domains
>2014-07-31 04:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-31 04:50:00,007 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 04:50:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 4:50 AM
>2014-07-31 04:50:00,023 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-31 04:50:00,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64111a43
>2014-07-31 04:51:26,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 04:51:26,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 04:51:26,912 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 04:51:26,913 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 04:52:27,002 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-07-31 04:52:27,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] FINISH, SpmStopVDSCommand, log id: 5f1fb654
>2014-07-31 04:52:27,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] FINISH, ResetIrsVDSCommand, log id: 1f761dbc
>2014-07-31 04:52:27,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 64111a43
>2014-07-31 04:52:27,027 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 4:52 AM
>2014-07-31 04:52:27,030 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [320efffb] Correlation ID: 320efffb, Job ID: 31346ad7-d12d-4841-b086-b78a4c76f327, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 04:52:27,038 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [519c108f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 04:52:27,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 04:52:27,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 04:52:27,129 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 04:52:27,136 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 04:52:27,144 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 04:52:27,157 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [9f67958] Correlation ID: 9f67958, Job ID: d85583c2-b16d-4e16-aa80-f0f342579705, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 04:52:27,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [9f67958] Removing vds [] from the domain in maintenance cache
>2014-07-31 04:52:27,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [9f67958] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 04:52:27,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 04:52:27,177 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 04:52:27,208 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 04:52:27,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Removing vds [] from the domain in maintenance cache
>2014-07-31 04:52:27,234 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 04:52:27,235 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [519c108f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 04:53:26,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 04:53:26,933 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 04:53:27,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-15) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-31 04:53:27,689 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-07-31 04:53:27,707 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-31 04:53:27,715 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-31 04:53:30,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5bfde9d9
>2014-07-31 04:53:30,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] FINISH, GetHardwareInfoVDSCommand, log id: 5bfde9d9
>2014-07-31 04:53:31,025 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [53909aaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 04:53:31,074 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [53909aaa] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-31 04:53:31,076 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [8936159] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 04:53:31,083 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 04:53:31,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6153fd97
>2014-07-31 04:53:31,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6153fd97
>2014-07-31 04:53:31,162 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Host rack storage connection was succeeded 
>2014-07-31 04:53:31,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 543fc72a
>2014-07-31 04:53:36,937 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 04:53:36,941 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 04:54:31,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 543fc72a
>2014-07-31 04:54:31,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1841609e
>2014-07-31 04:54:31,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1841609e
>2014-07-31 04:54:31,741 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 04:54:31,750 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 04:54:31,770 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 04:54:37,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 04:54:37,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 04:54:37,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 714e82ad
>2014-07-31 04:54:37,116 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling started: taskId = 5fa4711d-b918-4774-b314-552fbb728ca5
>2014-07-31 04:54:38,122 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling ended: taskId = 5fa4711d-b918-4774-b314-552fbb728ca5 task status = finished
>2014-07-31 04:54:38,178 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling ended, spm status: SPM
>2014-07-31 04:54:38,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5fa4711d-b918-4774-b314-552fbb728ca5), log id: 3ddcf217
>2014-07-31 04:54:38,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] FINISH, HSMClearTaskVDSCommand, log id: 3ddcf217
>2014-07-31 04:54:38,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e3b69c7, log id: 714e82ad
>2014-07-31 04:54:38,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 04:54:38,236 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 04:54:38,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 56049f5c
>2014-07-31 04:54:38,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 04:54:38,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a516f5
>2014-07-31 04:54:38,296 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a516f5
>2014-07-31 04:54:38,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 56049f5c
>2014-07-31 04:54:38,299 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] Discovered no tasks on Storage Pool Default
>2014-07-31 05:23:55,247 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Principal admin::internal synchronized
>2014-07-31 05:24:22,409 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf.
>2014-07-31 06:23:55,254 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-07-31 06:24:22,423 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf.
>2014-07-31 07:23:55,261 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-07-31 07:24:22,460 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf.
>2014-07-31 07:59:17,969 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 07:59:17,971 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 08:00:56,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 08:00:56,475 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 08:00:56,479 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [653adc38] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 08:00:56,481 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [653adc38] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 08:00:56,495 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [653adc38] Correlation ID: 653adc38, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 08:02:56,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 08:02:56,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [653adc38] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 08:02:56,538 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 08:02:56,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 08:02:56,566 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 08:02:56,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 688a1691
>2014-07-31 08:02:56,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1e2def37
>2014-07-31 08:02:56,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 08:04:56,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 08:04:56,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 08:04:56,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 08:04:56,521 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 08:05:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Autorecovering 1 storage domains
>2014-07-31 08:05:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-31 08:05:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 08:05:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 8:05 AM
>2014-07-31 08:05:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-31 08:05:00,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4079a353
>2014-07-31 08:05:56,598 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-07-31 08:05:56,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] FINISH, SpmStopVDSCommand, log id: 1e2def37
>2014-07-31 08:05:56,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] FINISH, ResetIrsVDSCommand, log id: 688a1691
>2014-07-31 08:05:56,615 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] Correlation ID: 7b3e6f1d, Job ID: 8f6c7c49-c618-4c4b-aaf3-76afbb87ea8f, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 08:05:56,620 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [653adc38] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 08:05:56,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 08:05:56,686 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 08:05:56,730 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 08:05:56,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Removing vds [] from the domain in maintenance cache
>2014-07-31 08:05:56,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 08:05:56,764 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [653adc38] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 08:05:56,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 08:05:56,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 08:05:56,785 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 08:05:56,794 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 08:05:56,800 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 08:05:56,811 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Correlation ID: 7f63a243, Job ID: 55f328a5-b0e3-4d78-b4e4-b688ed2e2e20, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 08:05:56,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Removing vds [] from the domain in maintenance cache
>2014-07-31 08:05:56,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 08:06:00,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4079a353
>2014-07-31 08:06:00,046 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 8:06 AM
>2014-07-31 08:06:56,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 08:06:56,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [4494a716] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 08:06:57,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-27) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-07-31 08:06:57,616 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-07-31 08:06:57,637 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-07-31 08:06:57,648 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-07-31 08:07:00,876 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 166134dd
>2014-07-31 08:07:00,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] FINISH, GetHardwareInfoVDSCommand, log id: 166134dd
>2014-07-31 08:07:00,982 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [41f51a04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 08:07:00,992 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [41f51a04] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-07-31 08:07:00,993 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [13b4a74d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 08:07:01,003 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 08:07:01,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a7ce43d
>2014-07-31 08:07:01,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3a7ce43d
>2014-07-31 08:07:01,078 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Host rack storage connection was succeeded 
>2014-07-31 08:07:01,084 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 68808caa
>2014-07-31 08:07:06,541 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 08:07:06,545 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 08:08:01,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, ConnectStoragePoolVDSCommand, log id: 68808caa
>2014-07-31 08:08:01,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5a2647d
>2014-07-31 08:08:01,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5a2647d
>2014-07-31 08:08:01,613 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 08:08:01,620 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 08:08:01,638 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 08:08:06,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 08:08:06,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 08:08:06,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 425c0c89
>2014-07-31 08:08:06,719 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = 4fd9b3ab-eccf-41fa-8086-a7eee492113b
>2014-07-31 08:08:07,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = 4fd9b3ab-eccf-41fa-8086-a7eee492113b task status = finished
>2014-07-31 08:08:07,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM
>2014-07-31 08:08:07,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4fd9b3ab-eccf-41fa-8086-a7eee492113b), log id: 3a621dfe
>2014-07-31 08:08:07,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 3a621dfe
>2014-07-31 08:08:07,870 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8dddd8e, log id: 425c0c89
>2014-07-31 08:08:07,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 08:08:07,881 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 08:08:07,882 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-12) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 08:08:07,895 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 666f039b
>2014-07-31 08:08:07,922 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 08:08:07,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23472fec
>2014-07-31 08:08:07,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23472fec
>2014-07-31 08:08:07,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 666f039b
>2014-07-31 08:08:07,931 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-12) Discovered no tasks on Storage Pool Default
>2014-07-31 08:23:55,267 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) Principal admin::internal synchronized
>2014-07-31 08:24:22,475 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) [341aec49] Attempting to update VMs/Templates Ovf.
>2014-07-31 08:38:03,596 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-07-31 08:38:03,621 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-31 08:38:32,640 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [12abc774] Running command: LogoutUserCommand internal: false.
>2014-07-31 08:38:32,646 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [12abc774] Correlation ID: 12abc774, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-07-31 09:23:55,273 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized
>2014-07-31 09:24:22,492 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Attempting to update VMs/Templates Ovf.
>2014-07-31 09:35:23,463 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-07-31 09:35:23,469 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-07-31 09:35:23,470 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-07-31 09:35:23,572 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-07-31 09:44:20,481 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-1) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 10:23:55,281 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-07-31 10:24:22,509 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf.
>2014-07-31 11:07:45,664 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 11:08:00,785 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 11:09:24,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 11:09:24,863 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 11:09:24,866 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 11:09:24,868 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 11:09:24,881 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Correlation ID: 7f1e369e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 11:11:24,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 11:11:24,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 11:11:24,914 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 11:11:24,915 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 11:11:24,951 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 11:11:24,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5b2f2794
>2014-07-31 11:11:24,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 590dc58f
>2014-07-31 11:11:24,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 11:13:24,881 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 11:13:24,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-3) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 11:13:24,887 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 11:13:24,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] FINISH, SpmStopVDSCommand, log id: 590dc58f
>2014-07-31 11:13:24,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 11:13:24,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 11:13:24,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] FINISH, ResetIrsVDSCommand, log id: 5b2f2794
>2014-07-31 11:13:24,888 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 11:13:24,908 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [385b441c] Correlation ID: 385b441c, Job ID: a6b4d423-858c-4f8d-94f8-b2e030d5a254, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 11:13:24,934 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 11:13:24,935 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [7f1e369e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 11:13:24,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 11:13:25,006 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 11:13:25,028 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [2015a55d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 11:13:25,035 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [2015a55d] Removing vds [] from the domain in maintenance cache
>2014-07-31 11:13:25,036 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [2015a55d] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 11:13:25,037 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [7f1e369e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 11:13:25,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 11:13:25,044 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 11:13:25,060 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 11:13:25,067 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 11:13:25,074 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 11:13:25,084 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Correlation ID: 20b596ad, Job ID: 56a226b7-1cb3-4ce0-bb3e-8dfde4b86066, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 11:13:25,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Removing vds [] from the domain in maintenance cache
>2014-07-31 11:13:25,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 11:13:28,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 108e0e8d
>2014-07-31 11:13:28,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] FINISH, GetHardwareInfoVDSCommand, log id: 108e0e8d
>2014-07-31 11:13:28,276 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [11c9633d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 11:13:28,327 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [2c688054] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 11:13:28,334 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 11:13:28,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: fe431fb
>2014-07-31 11:13:28,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: fe431fb
>2014-07-31 11:13:28,403 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Host rack storage connection was succeeded 
>2014-07-31 11:13:28,409 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2e70b61b
>2014-07-31 11:13:34,902 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 11:13:34,905 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 11:14:28,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, ConnectStoragePoolVDSCommand, log id: 2e70b61b
>2014-07-31 11:14:28,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2a421bf9
>2014-07-31 11:14:28,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2a421bf9
>2014-07-31 11:14:28,931 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 11:14:28,937 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 11:14:28,953 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [4341002] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 11:14:35,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 11:14:35,056 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 11:14:35,059 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 126eb4db
>2014-07-31 11:14:35,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling started: taskId = 84af9baf-f8cc-4ea7-bd5a-a859f5023d6e
>2014-07-31 11:14:36,114 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling ended: taskId = 84af9baf-f8cc-4ea7-bd5a-a859f5023d6e task status = finished
>2014-07-31 11:14:36,170 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling ended, spm status: SPM
>2014-07-31 11:14:36,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=84af9baf-f8cc-4ea7-bd5a-a859f5023d6e), log id: 2acc59e7
>2014-07-31 11:14:36,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] FINISH, HSMClearTaskVDSCommand, log id: 2acc59e7
>2014-07-31 11:14:36,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13b1b5ab, log id: 126eb4db
>2014-07-31 11:14:36,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 11:14:36,229 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 11:14:36,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41485ee4
>2014-07-31 11:14:36,274 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 11:14:36,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 55a33e6d
>2014-07-31 11:14:36,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55a33e6d
>2014-07-31 11:14:36,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41485ee4
>2014-07-31 11:14:36,284 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] Discovered no tasks on Storage Pool Default
>2014-07-31 11:23:55,288 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [341aec49] Principal admin::internal synchronized
>2014-07-31 11:24:22,523 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf.
>2014-07-31 12:23:55,297 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized
>2014-07-31 12:24:22,539 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf.
>2014-07-31 13:23:55,302 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Principal admin::internal synchronized
>2014-07-31 13:24:22,563 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) [52dee25c] Attempting to update VMs/Templates Ovf.
>2014-07-31 14:14:19,361 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 14:14:19,364 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 14:15:53,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 14:15:53,456 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 14:15:53,460 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 14:15:53,462 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 14:15:53,475 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [686e64a1] Correlation ID: 686e64a1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 14:17:53,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 14:17:53,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [686e64a1] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 14:17:53,505 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 14:17:53,506 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 14:17:53,553 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 14:17:53,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7b586e56
>2014-07-31 14:17:53,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 190eaae0
>2014-07-31 14:17:53,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 14:19:53,473 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 14:19:53,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] FINISH, SpmStopVDSCommand, log id: 190eaae0
>2014-07-31 14:19:53,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 14:19:53,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 14:19:53,480 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [5484a050] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 14:19:53,486 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 14:19:53,487 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [296dca52] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 14:19:53,489 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [296dca52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 14:19:53,494 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] FINISH, ResetIrsVDSCommand, log id: 7b586e56
>2014-07-31 14:19:53,522 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 14:19:53,526 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [cd347f3] Correlation ID: cd347f3, Job ID: b7387082-f3da-4de5-8add-f976dc51cf8e, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 14:19:53,532 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [686e64a1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 14:19:53,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 14:19:53,588 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 14:19:53,605 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 14:19:53,611 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 14:19:53,616 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 14:19:53,624 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Correlation ID: 5448ed13, Job ID: adaec078-f569-4930-b790-6fbfbf7e4a6e, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 14:19:53,629 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Removing vds [] from the domain in maintenance cache
>2014-07-31 14:19:53,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 14:19:53,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 14:19:53,636 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 14:19:53,655 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [63924f71] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 14:19:53,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [63924f71] Removing vds [] from the domain in maintenance cache
>2014-07-31 14:19:53,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [63924f71] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 14:19:53,664 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [686e64a1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 14:19:56,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 485a9c02
>2014-07-31 14:19:56,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, GetHardwareInfoVDSCommand, log id: 485a9c02
>2014-07-31 14:19:56,856 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [7eb60aa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 14:19:56,906 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-42) [3585d0b5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 14:19:56,914 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 14:19:56,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b720444
>2014-07-31 14:19:56,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7b720444
>2014-07-31 14:19:56,989 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Host rack storage connection was succeeded 
>2014-07-31 14:19:56,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 33616302
>2014-07-31 14:20:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering 2 storage domains
>2014-07-31 14:20:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-31 14:20:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 14:20:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 2:20 PM
>2014-07-31 14:20:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 2:20 PM
>2014-07-31 14:20:00,016 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [7390d6d6] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-07-31 14:20:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 14:20:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 2:20 PM
>2014-07-31 14:20:00,024 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 2:20 PM
>2014-07-31 14:20:03,487 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 14:20:03,491 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 14:20:57,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 33616302
>2014-07-31 14:20:57,392 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: d100a7a
>2014-07-31 14:20:57,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] FINISH, SetMOMPolicyParametersVDSCommand, log id: d100a7a
>2014-07-31 14:20:57,493 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 14:20:57,499 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 14:20:57,517 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 14:21:03,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 14:21:03,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 14:21:03,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35f8090e
>2014-07-31 14:21:03,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling started: taskId = fbecd537-eb0e-42d0-aa4b-0711a325814f
>2014-07-31 14:21:04,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling ended: taskId = fbecd537-eb0e-42d0-aa4b-0711a325814f task status = finished
>2014-07-31 14:21:04,730 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling ended, spm status: SPM
>2014-07-31 14:21:04,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-94) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fbecd537-eb0e-42d0-aa4b-0711a325814f), log id: 2ba224ca
>2014-07-31 14:21:04,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, HSMClearTaskVDSCommand, log id: 2ba224ca
>2014-07-31 14:21:04,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d28b13d, log id: 35f8090e
>2014-07-31 14:21:04,784 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 14:21:04,791 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 14:21:04,792 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 14:21:04,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2edfc7aa
>2014-07-31 14:21:04,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 14:21:04,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 44cb155e
>2014-07-31 14:21:04,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44cb155e
>2014-07-31 14:21:04,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2edfc7aa
>2014-07-31 14:21:04,840 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) Discovered no tasks on Storage Pool Default
>2014-07-31 14:23:55,310 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized
>2014-07-31 14:24:22,578 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-07-31 15:23:55,316 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [3adf532b] Principal admin::internal synchronized
>2014-07-31 15:24:22,595 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [375fcc6] Attempting to update VMs/Templates Ovf.
>2014-07-31 16:23:55,321 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Principal admin::internal synchronized
>2014-07-31 16:24:22,610 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf.
>2014-07-31 17:20:51,989 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 17:21:07,108 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 17:22:28,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 17:22:28,458 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 17:22:28,461 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [5e890dae] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 17:22:28,463 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [5e890dae] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 17:22:28,475 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [5e890dae] Correlation ID: 5e890dae, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 17:23:55,327 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized
>2014-07-31 17:24:22,638 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-07-31 17:24:28,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 17:24:28,495 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [5e890dae] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 17:24:28,496 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 17:24:28,497 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 17:24:28,544 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 17:24:28,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5a64bf32
>2014-07-31 17:24:28,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 394d37e4
>2014-07-31 17:24:28,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 17:25:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Autorecovering 1 storage domains
>2014-07-31 17:25:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-07-31 17:25:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 17:25:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 5:25 PM
>2014-07-31 17:25:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-07-31 17:25:00,032 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5467fea1
>2014-07-31 17:26:28,462 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 17:26:28,466 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] FINISH, SpmStopVDSCommand, log id: 394d37e4
>2014-07-31 17:26:28,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 17:26:28,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] FINISH, ConnectStorageServerVDSCommand, log id: 5467fea1
>2014-07-31 17:26:28,476 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-07-31 17:26:28,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 17:26:28,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 17:26:28,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 17:26:28,485 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 17:26:28,486 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 17:26:28,488 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 17:26:28,495 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-07-31 17:26:28,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] FINISH, ResetIrsVDSCommand, log id: 5a64bf32
>2014-07-31 17:26:28,518 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 5:26 PM
>2014-07-31 17:26:28,525 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 17:26:28,529 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [338c4d71] Correlation ID: 338c4d71, Job ID: b98a97ea-2aed-4bc8-97b9-9617c02701bd, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 17:26:28,534 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [5e890dae] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 17:26:28,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 17:26:28,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 17:26:28,643 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [357adb45] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 17:26:28,651 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [357adb45] Removing vds [] from the domain in maintenance cache
>2014-07-31 17:26:28,652 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [357adb45] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 17:26:28,653 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [5e890dae] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 17:26:28,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 17:26:28,660 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 17:26:28,675 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 17:26:28,681 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 17:26:28,688 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 17:26:28,697 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Correlation ID: 6e389c5f, Job ID: 4c6d13de-dacf-4ca0-80e2-15476848fd76, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 17:26:28,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Removing vds [] from the domain in maintenance cache
>2014-07-31 17:26:28,703 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 17:26:31,787 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2b3d491f
>2014-07-31 17:26:31,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 2b3d491f
>2014-07-31 17:26:31,865 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [6bb078c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 17:26:31,917 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [6e042952] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 17:26:31,924 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 17:26:31,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 45b9546b
>2014-07-31 17:26:31,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 45b9546b
>2014-07-31 17:26:31,996 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] Host rack storage connection was succeeded 
>2014-07-31 17:26:32,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 672731ca
>2014-07-31 17:26:38,485 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 17:26:38,489 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 17:27:32,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 672731ca
>2014-07-31 17:27:32,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 736ea1da
>2014-07-31 17:27:32,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] FINISH, SetMOMPolicyParametersVDSCommand, log id: 736ea1da
>2014-07-31 17:27:32,690 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [4763ce07] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 17:27:32,697 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [4763ce07] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 17:27:32,712 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 17:27:38,612 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 17:27:38,617 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 17:27:38,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72b227e4
>2014-07-31 17:27:38,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling started: taskId = ed8dfde7-0eef-42b2-a89d-1cf15eed5898
>2014-07-31 17:27:39,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling ended: taskId = ed8dfde7-0eef-42b2-a89d-1cf15eed5898 task status = finished
>2014-07-31 17:27:39,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling ended, spm status: SPM
>2014-07-31 17:27:39,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-5) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=ed8dfde7-0eef-42b2-a89d-1cf15eed5898), log id: 63ebc3a9
>2014-07-31 17:27:39,806 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, HSMClearTaskVDSCommand, log id: 63ebc3a9
>2014-07-31 17:27:39,808 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@402b2f44, log id: 72b227e4
>2014-07-31 17:27:39,812 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 17:27:39,819 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 17:27:39,820 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-16) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 17:27:39,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d4b4cef
>2014-07-31 17:27:39,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 17:27:39,891 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3c2caa8c
>2014-07-31 17:27:39,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c2caa8c
>2014-07-31 17:27:39,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d4b4cef
>2014-07-31 17:27:39,898 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default
>2014-07-31 18:23:55,335 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized
>2014-07-31 18:24:22,640 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf.
>2014-07-31 19:23:55,342 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized
>2014-07-31 19:24:22,681 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [7526d482] Attempting to update VMs/Templates Ovf.
>2014-07-31 20:23:55,348 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-07-31 20:24:22,696 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
>2014-07-31 20:27:27,645 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 20:27:27,647 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 20:28:56,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 20:28:56,878 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 20:28:56,882 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 20:28:56,885 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 20:28:56,901 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Correlation ID: 1cb6c42c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 20:30:56,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 20:30:56,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 20:30:56,929 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 20:30:56,930 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 20:30:56,969 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 20:30:56,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2de22ba1
>2014-07-31 20:30:56,994 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 907165d
>2014-07-31 20:30:57,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 20:32:56,897 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 20:32:56,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 20:32:56,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] FINISH, SpmStopVDSCommand, log id: 907165d
>2014-07-31 20:32:56,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 20:32:56,931 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] FINISH, ResetIrsVDSCommand, log id: 2de22ba1
>2014-07-31 20:32:56,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 20:32:56,942 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [5484a050] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 20:32:56,947 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [5484a050] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 20:32:56,961 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [756dc666] Correlation ID: 756dc666, Job ID: 3d837794-9b23-4767-b9ee-fd56bc9befe3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 20:32:57,023 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [1cb6c42c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 20:32:57,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 20:32:57,032 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [5484a050] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 20:32:57,095 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 20:32:57,113 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 20:32:57,120 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] Removing vds [] from the domain in maintenance cache
>2014-07-31 20:32:57,122 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 20:32:57,124 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [1cb6c42c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 20:32:57,128 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 20:32:57,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 20:32:57,151 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 20:32:57,161 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 20:32:57,169 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 20:32:57,180 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [65949611] Correlation ID: 65949611, Job ID: c68098ca-bbe7-4bc9-8c1a-aed4a93b4c36, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 20:32:57,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [65949611] Removing vds [] from the domain in maintenance cache
>2014-07-31 20:32:57,186 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [65949611] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 20:33:00,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 20e31637
>2014-07-31 20:33:00,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] FINISH, GetHardwareInfoVDSCommand, log id: 20e31637
>2014-07-31 20:33:00,377 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [3da73ce0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 20:33:00,426 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [7af5412a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 20:33:00,435 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 20:33:00,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7a212569
>2014-07-31 20:33:00,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7a212569
>2014-07-31 20:33:00,508 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] Host rack storage connection was succeeded 
>2014-07-31 20:33:00,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 17842b92
>2014-07-31 20:33:06,948 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 20:33:06,952 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 20:34:00,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 17842b92
>2014-07-31 20:34:00,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6ec70026
>2014-07-31 20:34:01,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6ec70026
>2014-07-31 20:34:01,059 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [732318f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 20:34:01,066 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [732318f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 20:34:01,081 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 20:34:07,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 20:34:07,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 20:34:07,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40095834
>2014-07-31 20:34:07,128 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling started: taskId = cb2789cd-5a2d-402a-af18-fdd99a3ee7bf
>2014-07-31 20:34:08,135 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended: taskId = cb2789cd-5a2d-402a-af18-fdd99a3ee7bf task status = finished
>2014-07-31 20:34:08,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended, spm status: SPM
>2014-07-31 20:34:08,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cb2789cd-5a2d-402a-af18-fdd99a3ee7bf), log id: 327a561b
>2014-07-31 20:34:08,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, HSMClearTaskVDSCommand, log id: 327a561b
>2014-07-31 20:34:08,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1a59393c, log id: 40095834
>2014-07-31 20:34:08,245 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 20:34:08,253 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1b83a377] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 20:34:08,254 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-36) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-07-31 20:34:08,264 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6120e90a
>2014-07-31 20:34:08,296 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 20:34:08,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5b4a57ff
>2014-07-31 20:34:08,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b4a57ff
>2014-07-31 20:34:08,304 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6120e90a
>2014-07-31 20:34:08,305 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-36) [1b83a377] Discovered no tasks on Storage Pool Default
>2014-07-31 21:23:55,353 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-07-31 21:24:22,724 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-07-31 22:23:55,362 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-07-31 22:24:22,741 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Attempting to update VMs/Templates Ovf.
>2014-07-31 23:23:55,369 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-07-31 23:24:22,758 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf.
>2014-07-31 23:33:47,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-07-31 23:34:02,279 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-07-31 23:35:24,483 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 23:35:24,487 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 23:35:24,491 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-07-31 23:35:24,493 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-07-31 23:35:24,507 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [37510499] Correlation ID: 37510499, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-07-31 23:37:24,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 23:37:24,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 23:37:24,540 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [37510499] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-07-31 23:37:24,540 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-07-31 23:37:24,590 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-07-31 23:37:24,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 24b0bb1d
>2014-07-31 23:37:24,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3c1908f0
>2014-07-31 23:37:24,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-07-31 23:39:24,509 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-07-31 23:39:24,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] FINISH, SpmStopVDSCommand, log id: 3c1908f0
>2014-07-31 23:39:24,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-07-31 23:39:24,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-07-31 23:39:24,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-07-31 23:39:24,515 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-07-31 23:39:24,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] FINISH, ResetIrsVDSCommand, log id: 24b0bb1d
>2014-07-31 23:39:24,520 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-07-31 23:39:24,540 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] Correlation ID: 28c5ee0f, Job ID: 0628d3f4-ca61-4c68-a7c1-2cd663589ca1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-07-31 23:39:24,559 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-07-31 23:39:24,559 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [37510499] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 23:39:24,562 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-07-31 23:39:24,617 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-07-31 23:39:24,636 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [7235e194] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-07-31 23:39:24,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [7235e194] Removing vds [] from the domain in maintenance cache
>2014-07-31 23:39:24,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [7235e194] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 23:39:24,645 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [37510499] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-07-31 23:39:24,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-07-31 23:39:24,652 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-07-31 23:39:24,667 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 23:39:24,673 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-07-31 23:39:24,679 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-07-31 23:39:24,689 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [323a2598] Correlation ID: 323a2598, Job ID: e7707ab5-d1d9-4555-8581-8f995aead2dc, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-07-31 23:39:24,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [323a2598] Removing vds [] from the domain in maintenance cache
>2014-07-31 23:39:24,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [323a2598] Removing host(s) [] from hosts unseen domain report cache
>2014-07-31 23:39:27,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 651073b7
>2014-07-31 23:39:27,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, GetHardwareInfoVDSCommand, log id: 651073b7
>2014-07-31 23:39:27,904 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [676113c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 23:39:27,954 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-18) [2bda1b13] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 23:39:27,963 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-07-31 23:39:27,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 76460e8f
>2014-07-31 23:39:28,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 76460e8f
>2014-07-31 23:39:28,034 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Host rack storage connection was succeeded 
>2014-07-31 23:39:28,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2eb80d8f
>2014-07-31 23:39:34,522 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 23:39:34,527 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-07-31 23:40:28,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 2eb80d8f
>2014-07-31 23:40:28,452 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 59540fe9
>2014-07-31 23:40:28,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 59540fe9
>2014-07-31 23:40:28,554 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-07-31 23:40:28,560 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-07-31 23:40:28,576 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-07-31 23:40:34,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-07-31 23:40:34,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-07-31 23:40:34,683 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b3b1dcb
>2014-07-31 23:40:34,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = 169337fe-d0af-4239-8f2c-cfd8e87ae4da
>2014-07-31 23:40:35,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = 169337fe-d0af-4239-8f2c-cfd8e87ae4da task status = finished
>2014-07-31 23:40:35,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM
>2014-07-31 23:40:35,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=169337fe-d0af-4239-8f2c-cfd8e87ae4da), log id: 19538009
>2014-07-31 23:40:35,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 19538009
>2014-07-31 23:40:35,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2dae15f9, log id: 5b3b1dcb
>2014-07-31 23:40:35,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208
>2014-07-31 23:40:35,855 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-07-31 23:40:35,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6d97f318
>2014-07-31 23:40:35,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-07-31 23:40:35,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16fa17c3
>2014-07-31 23:40:35,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16fa17c3
>2014-07-31 23:40:35,909 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d97f318
>2014-07-31 23:40:35,910 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) [4760e42] Discovered no tasks on Storage Pool Default
>2014-08-01 00:23:55,376 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) [5497a121] Principal admin::internal synchronized
>2014-08-01 00:24:22,794 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf.
>2014-08-01 01:23:55,384 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Principal admin::internal synchronized
>2014-08-01 01:24:22,827 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf.
>2014-08-01 02:23:55,390 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized
>2014-08-01 02:24:22,842 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-01 02:40:20,402 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 02:40:20,405 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 02:41:50,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 02:41:50,248 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 02:41:50,250 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-77) [2e3238db] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 02:41:50,257 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-77) [2e3238db] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 02:41:50,270 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [2e3238db] Correlation ID: 2e3238db, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 02:43:50,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 02:43:50,298 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2e3238db] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 02:43:50,299 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 02:43:50,300 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 02:43:50,354 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 02:43:50,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 36ad7aa7
>2014-08-01 02:43:50,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 415ce57b
>2014-08-01 02:43:50,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 02:45:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) Autorecovering 1 storage domains
>2014-08-01 02:45:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-01 02:45:00,007 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 02:45:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 2:45 AM
>2014-08-01 02:45:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-01 02:45:00,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1127d8cd
>2014-08-01 02:45:50,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 02:45:50,272 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 02:45:50,273 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 02:45:50,274 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 02:46:50,273 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 02:46:50,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 02:46:50,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-5) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 02:46:50,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 02:46:50,290 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 02:46:50,286 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 02:46:50,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] FINISH, SpmStopVDSCommand, log id: 415ce57b
>2014-08-01 02:46:50,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] FINISH, ConnectStorageServerVDSCommand, log id: 1127d8cd
>2014-08-01 02:46:50,303 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-01 02:46:50,296 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 02:46:50,317 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-01 02:46:50,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] FINISH, ResetIrsVDSCommand, log id: 36ad7aa7
>2014-08-01 02:46:50,339 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 2:46 AM
>2014-08-01 02:46:50,346 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 02:46:50,352 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [d926c49] Correlation ID: d926c49, Job ID: a6e60d1e-c692-417d-9a01-bb212d2eea4e, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 02:46:50,360 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [2e3238db] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 02:46:50,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 02:46:50,432 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 02:46:50,455 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 02:46:50,462 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 02:46:50,469 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 02:46:50,479 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Correlation ID: 5e1e7b65, Job ID: bb9449be-55e4-4cdc-a79d-5bfecfab29a5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 02:46:50,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Removing vds [] from the domain in maintenance cache
>2014-08-01 02:46:50,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 02:46:50,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 02:46:50,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 02:46:50,518 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [312d0f0] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 02:46:50,525 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [312d0f0] Removing vds [] from the domain in maintenance cache
>2014-08-01 02:46:50,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [312d0f0] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 02:46:50,527 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [2e3238db] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 02:46:53,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 513f447e
>2014-08-01 02:46:53,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] FINISH, GetHardwareInfoVDSCommand, log id: 513f447e
>2014-08-01 02:46:53,677 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [230b5395] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 02:46:53,689 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [5a39e60a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 02:46:53,697 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 02:46:53,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70a08d8
>2014-08-01 02:46:53,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 70a08d8
>2014-08-01 02:46:53,770 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Host rack storage connection was succeeded 
>2014-08-01 02:46:53,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 38974dad
>2014-08-01 02:47:00,302 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 02:47:00,306 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 02:47:54,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 38974dad
>2014-08-01 02:47:54,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7aed3c4
>2014-08-01 02:47:54,266 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7aed3c4
>2014-08-01 02:47:54,273 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 02:47:54,279 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 02:47:54,301 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [2809936b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 02:48:00,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 02:48:00,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 02:48:00,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e2778e9
>2014-08-01 02:48:00,483 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling started: taskId = 21499035-0c4b-4878-9f26-1840bd46bda9
>2014-08-01 02:48:01,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling ended: taskId = 21499035-0c4b-4878-9f26-1840bd46bda9 task status = finished
>2014-08-01 02:48:01,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling ended, spm status: SPM
>2014-08-01 02:48:01,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=21499035-0c4b-4878-9f26-1840bd46bda9), log id: 59174e32
>2014-08-01 02:48:01,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, HSMClearTaskVDSCommand, log id: 59174e32
>2014-08-01 02:48:01,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bb3bf0e, log id: 3e2778e9
>2014-08-01 02:48:01,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 02:48:01,606 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 02:48:01,608 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 02:48:01,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 68b90328
>2014-08-01 02:48:01,647 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 02:48:01,649 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 25cb6a90
>2014-08-01 02:48:01,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25cb6a90
>2014-08-01 02:48:01,655 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68b90328
>2014-08-01 02:48:01,656 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] Discovered no tasks on Storage Pool Default
>2014-08-01 03:23:55,396 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) [56f1581c] Principal admin::internal synchronized
>2014-08-01 03:24:22,857 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-08-01 03:35:35,000 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Start deleteAgedOutAuditLogs
>2014-08-01 03:35:35,005 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Finished deleteAgedOutAuditLogs
>2014-08-01 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Start deleteAgedOutCommandEntities
>2014-08-01 03:35:35,012 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Finished deleteAgedOutCommandEntities
>2014-08-01 03:55:15,578 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 04:23:55,401 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Principal admin::internal synchronized
>2014-08-01 04:24:22,887 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-08-01 05:23:55,411 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) [37510499] Principal admin::internal synchronized
>2014-08-01 05:24:22,906 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Attempting to update VMs/Templates Ovf.
>2014-08-01 05:47:37,182 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 05:47:37,184 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 05:49:16,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 05:49:16,288 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 05:49:16,292 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [161c5527] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 05:49:16,295 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [161c5527] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 05:49:16,309 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [161c5527] Correlation ID: 161c5527, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 05:51:16,335 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 05:51:16,338 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) [161c5527] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 05:51:16,338 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 05:51:16,339 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 05:51:16,392 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 05:51:16,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6b16e474
>2014-08-01 05:51:16,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2398c319
>2014-08-01 05:51:16,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 05:53:16,306 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 05:53:16,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] FINISH, SpmStopVDSCommand, log id: 2398c319
>2014-08-01 05:53:16,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 05:53:16,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 05:53:16,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] FINISH, ResetIrsVDSCommand, log id: 6b16e474
>2014-08-01 05:53:16,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-49) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 05:53:16,318 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 05:53:16,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 05:53:16,332 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [665db16f] Correlation ID: 665db16f, Job ID: cafbb138-6b38-420f-a478-117c34d80f15, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 05:53:16,353 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 05:53:16,354 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [161c5527] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 05:53:16,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 05:53:16,425 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 05:53:16,444 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 05:53:16,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] Removing vds [] from the domain in maintenance cache
>2014-08-01 05:53:16,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 05:53:16,454 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [161c5527] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 05:53:16,456 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 05:53:16,461 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 05:53:16,477 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 05:53:16,490 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 05:53:16,496 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 05:53:16,515 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Correlation ID: 3ddd3b82, Job ID: 43414010-2dea-431c-9147-65f9fd005f0d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 05:53:16,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Removing vds [] from the domain in maintenance cache
>2014-08-01 05:53:16,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 05:53:19,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5857efb6
>2014-08-01 05:53:19,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] FINISH, GetHardwareInfoVDSCommand, log id: 5857efb6
>2014-08-01 05:53:19,708 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [216d7974] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 05:53:19,757 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [c701568] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 05:53:19,766 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 05:53:19,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 96f1342
>2014-08-01 05:53:19,847 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 96f1342
>2014-08-01 05:53:19,849 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] Host rack storage connection was succeeded 
>2014-08-01 05:53:19,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 78694d3f
>2014-08-01 05:53:26,319 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-18) [5497a121] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 05:53:26,322 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-18) [5497a121] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 05:54:20,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 78694d3f
>2014-08-01 05:54:20,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 403faecb
>2014-08-01 05:54:20,387 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 403faecb
>2014-08-01 05:54:20,394 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [feae1d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 05:54:20,400 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [feae1d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 05:54:20,420 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 05:54:26,437 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 05:54:26,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 05:54:26,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4291af7a
>2014-08-01 05:54:26,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling started: taskId = 10b3c802-bda7-4ea0-81d5-f467afdb3e90
>2014-08-01 05:54:27,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling ended: taskId = 10b3c802-bda7-4ea0-81d5-f467afdb3e90 task status = finished
>2014-08-01 05:54:27,588 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling ended, spm status: SPM
>2014-08-01 05:54:27,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=10b3c802-bda7-4ea0-81d5-f467afdb3e90), log id: 6644cbea
>2014-08-01 05:54:27,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, HSMClearTaskVDSCommand, log id: 6644cbea
>2014-08-01 05:54:27,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60323fca, log id: 4291af7a
>2014-08-01 05:54:27,641 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 05:54:27,648 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 05:54:27,649 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 05:54:27,667 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d279b1f
>2014-08-01 05:54:27,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 05:54:27,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 615f9f81
>2014-08-01 05:54:27,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 615f9f81
>2014-08-01 05:54:27,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d279b1f
>2014-08-01 05:54:27,708 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] Discovered no tasks on Storage Pool Default
>2014-08-01 06:23:55,417 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-45) [51d0e706] Principal admin::internal synchronized
>2014-08-01 06:24:22,936 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-08-01 07:23:55,423 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Principal admin::internal synchronized
>2014-08-01 07:24:22,951 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf.
>2014-08-01 08:23:55,432 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-01 08:24:22,981 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf.
>2014-08-01 08:54:12,367 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 08:54:12,369 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 08:55:43,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 08:55:43,237 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 08:55:43,245 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 08:55:43,249 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 08:55:43,266 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Correlation ID: 156fe0ca, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 08:57:43,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 08:57:43,295 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [156fe0ca] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 08:57:43,296 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 08:57:43,298 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 08:57:43,333 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 08:57:43,350 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 66ddc035
>2014-08-01 08:57:43,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5c0aff22
>2014-08-01 08:57:43,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 08:59:43,257 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 08:59:43,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] FINISH, SpmStopVDSCommand, log id: 5c0aff22
>2014-08-01 08:59:43,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-27) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 08:59:43,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 08:59:43,266 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 08:59:43,266 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] FINISH, ResetIrsVDSCommand, log id: 66ddc035
>2014-08-01 08:59:43,265 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 08:59:43,270 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 08:59:43,282 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [726a0f53] Correlation ID: 726a0f53, Job ID: 94ff1dbb-8864-493e-a5ee-83729ef01419, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 08:59:43,302 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 08:59:43,304 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [156fe0ca] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 08:59:43,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 08:59:43,373 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 08:59:43,389 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 08:59:43,397 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 08:59:43,402 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 08:59:43,411 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Correlation ID: 7dac8714, Job ID: 44835dc1-8665-46e7-a0be-79bd9f034e2b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 08:59:43,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Removing vds [] from the domain in maintenance cache
>2014-08-01 08:59:43,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 08:59:43,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 08:59:43,425 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 08:59:43,443 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 08:59:43,452 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] Removing vds [] from the domain in maintenance cache
>2014-08-01 08:59:43,453 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 08:59:43,453 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [156fe0ca] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 08:59:46,562 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3a9e57f0
>2014-08-01 08:59:46,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, GetHardwareInfoVDSCommand, log id: 3a9e57f0
>2014-08-01 08:59:46,645 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [39e62ae6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 08:59:46,657 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-67) [25a43784] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 08:59:46,665 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 08:59:46,677 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2f00b828
>2014-08-01 08:59:46,736 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 2f00b828
>2014-08-01 08:59:46,738 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] Host rack storage connection was succeeded 
>2014-08-01 08:59:46,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 39b166f0
>2014-08-01 08:59:53,272 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 08:59:53,275 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 09:00:47,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 39b166f0
>2014-08-01 09:00:47,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: c8d4332
>2014-08-01 09:00:47,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] FINISH, SetMOMPolicyParametersVDSCommand, log id: c8d4332
>2014-08-01 09:00:47,290 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [13f100ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 09:00:47,296 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [13f100ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 09:00:47,321 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [6199900c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 09:00:53,422 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 09:00:53,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 09:00:53,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 544ea85a
>2014-08-01 09:00:53,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = 58f02b46-9ba6-42d7-8910-03b6ca4bc88e
>2014-08-01 09:00:54,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = 58f02b46-9ba6-42d7-8910-03b6ca4bc88e task status = finished
>2014-08-01 09:00:54,542 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM
>2014-08-01 09:00:54,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=58f02b46-9ba6-42d7-8910-03b6ca4bc88e), log id: 3eec0a34
>2014-08-01 09:00:54,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 3eec0a34
>2014-08-01 09:00:54,590 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ed410e8, log id: 544ea85a
>2014-08-01 09:00:54,595 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 09:00:54,601 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 09:00:54,611 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3f91f119
>2014-08-01 09:00:54,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 09:00:54,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1c10698
>2014-08-01 09:00:54,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c10698
>2014-08-01 09:00:54,654 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f91f119
>2014-08-01 09:00:54,655 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default
>2014-08-01 09:23:55,439 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-01 09:24:22,996 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) [797bf730] Attempting to update VMs/Templates Ovf.
>2014-08-01 10:23:55,446 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) Principal admin::internal synchronized
>2014-08-01 10:24:23,016 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf.
>2014-08-01 11:23:55,452 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-100) [52a75f18] Principal admin::internal synchronized
>2014-08-01 11:24:23,031 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) [37510499] Attempting to update VMs/Templates Ovf.
>2014-08-01 12:00:44,899 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 12:01:00,020 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 12:02:15,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 12:02:15,463 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 12:02:15,466 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 12:02:15,469 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 12:02:15,481 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Correlation ID: 18f9dc27, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 12:04:15,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 12:04:15,516 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [18f9dc27] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 12:04:15,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 12:04:15,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 12:04:15,590 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 12:04:15,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2e2d7f1c
>2014-08-01 12:04:15,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 16618ea
>2014-08-01 12:04:15,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 12:05:00,014 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering 1 storage domains
>2014-08-01 12:05:00,016 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-01 12:05:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 12:05:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 12:05 PM
>2014-08-01 12:05:00,034 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-01 12:05:00,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a2232aa
>2014-08-01 12:06:15,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 12:06:15,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-52) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 12:06:15,495 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 12:06:15,488 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 12:06:15,499 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] FINISH, SpmStopVDSCommand, log id: 16618ea
>2014-08-01 12:06:15,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 12:06:15,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 12:06:15,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] FINISH, ConnectStorageServerVDSCommand, log id: 3a2232aa
>2014-08-01 12:06:15,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] FINISH, ResetIrsVDSCommand, log id: 2e2d7f1c
>2014-08-01 12:06:15,497 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 12:06:15,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] Correlation ID: 38a1f3c4, Job ID: 5d187e6b-dfcf-48bc-af17-d723a1138731, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 12:06:15,510 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-01 12:06:15,541 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-01 12:06:15,543 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 12:06 PM
>2014-08-01 12:06:15,543 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-32) [18f9dc27] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 12:06:15,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 12:06:15,547 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 12:06:15,610 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 12:06:15,637 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-36) [a43af] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 12:06:15,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [a43af] Removing vds [] from the domain in maintenance cache
>2014-08-01 12:06:15,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [a43af] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 12:06:15,647 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-32) [18f9dc27] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 12:06:15,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 12:06:15,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 12:06:15,673 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 12:06:15,682 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 12:06:15,693 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 12:06:15,710 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [7951b282] Correlation ID: 7951b282, Job ID: 2be04eff-7eb2-448f-88d4-e7fc1c1b7628, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 12:06:15,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [7951b282] Removing vds [] from the domain in maintenance cache
>2014-08-01 12:06:15,720 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [7951b282] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 12:06:19,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4191f0b4
>2014-08-01 12:06:19,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, GetHardwareInfoVDSCommand, log id: 4191f0b4
>2014-08-01 12:06:19,122 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [97e99d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 12:06:19,172 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [5b66a69e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 12:06:19,180 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 12:06:19,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43da23a9
>2014-08-01 12:06:19,248 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 43da23a9
>2014-08-01 12:06:19,249 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] Host rack storage connection was succeeded 
>2014-08-01 12:06:19,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1c6f2a34
>2014-08-01 12:06:25,511 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 12:06:25,515 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 12:07:19,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 1c6f2a34
>2014-08-01 12:07:19,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: a44339e
>2014-08-01 12:07:19,801 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] FINISH, SetMOMPolicyParametersVDSCommand, log id: a44339e
>2014-08-01 12:07:19,811 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7333b236] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 12:07:19,816 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7333b236] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 12:07:19,834 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 12:07:25,635 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 12:07:25,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 12:07:25,642 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f1640e8
>2014-08-01 12:07:25,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling started: taskId = 5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67
>2014-08-01 12:07:26,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended: taskId = 5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67 task status = finished
>2014-08-01 12:07:26,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended, spm status: SPM
>2014-08-01 12:07:26,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67), log id: 7d684184
>2014-08-01 12:07:26,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, HSMClearTaskVDSCommand, log id: 7d684184
>2014-08-01 12:07:26,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a058eaf, log id: 4f1640e8
>2014-08-01 12:07:26,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 12:07:26,814 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 12:07:26,816 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-49) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 12:07:26,829 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3aaec3e
>2014-08-01 12:07:26,855 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 12:07:26,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6c15f4d3
>2014-08-01 12:07:26,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c15f4d3
>2014-08-01 12:07:26,863 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3aaec3e
>2014-08-01 12:07:26,864 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) Discovered no tasks on Storage Pool Default
>2014-08-01 12:23:55,460 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) [797bf730] Principal admin::internal synchronized
>2014-08-01 12:24:23,044 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf.
>2014-08-01 12:44:10,749 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-43) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 13:23:55,471 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-01 13:24:23,078 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Attempting to update VMs/Templates Ovf.
>2014-08-01 14:23:55,478 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [375fcc6] Principal admin::internal synchronized
>2014-08-01 14:24:23,092 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) [52d379dc] Attempting to update VMs/Templates Ovf.
>2014-08-01 15:07:05,529 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 15:07:20,660 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 15:08:41,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 15:08:41,974 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 15:08:41,977 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 15:08:41,979 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 15:08:41,998 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Correlation ID: 78e7d47b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 15:10:42,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 15:10:42,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [78e7d47b] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 15:10:42,027 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 15:10:42,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 15:10:42,057 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 15:10:42,074 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4eb880d7
>2014-08-01 15:10:42,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 26d2a09
>2014-08-01 15:10:42,083 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 15:12:41,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 15:12:41,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 15:12:41,998 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 15:12:41,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 15:12:42,004 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 15:12:42,005 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 15:12:42,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] FINISH, SpmStopVDSCommand, log id: 26d2a09
>2014-08-01 15:12:42,006 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 15:12:42,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] FINISH, ResetIrsVDSCommand, log id: 4eb880d7
>2014-08-01 15:12:42,035 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 15:12:42,042 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24a49849] Correlation ID: 24a49849, Job ID: 5e3f6925-8b9b-4940-86ee-f3972c1a690a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 15:12:42,047 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [78e7d47b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 15:12:42,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 15:12:42,117 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 15:12:42,137 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [29a8744c] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 15:12:42,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [29a8744c] Removing vds [] from the domain in maintenance cache
>2014-08-01 15:12:42,144 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [29a8744c] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 15:12:42,145 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [78e7d47b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 15:12:42,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 15:12:42,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 15:12:42,167 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 15:12:42,174 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 15:12:42,179 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 15:12:42,188 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Correlation ID: 3b8d5173, Job ID: d045c7e4-638c-45f9-af29-f699a4e1e590, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 15:12:42,193 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Removing vds [] from the domain in maintenance cache
>2014-08-01 15:12:42,194 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 15:12:45,293 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 60b6d538
>2014-08-01 15:12:45,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] FINISH, GetHardwareInfoVDSCommand, log id: 60b6d538
>2014-08-01 15:12:45,385 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [26f56f7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 15:12:45,397 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [28ec6d70] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 15:12:45,405 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 15:12:45,412 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4a930483
>2014-08-01 15:12:45,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4a930483
>2014-08-01 15:12:45,472 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] Host rack storage connection was succeeded 
>2014-08-01 15:12:45,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5225e95b
>2014-08-01 15:12:52,001 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 15:12:52,005 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 15:13:45,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 5225e95b
>2014-08-01 15:13:45,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45e0616
>2014-08-01 15:13:45,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45e0616
>2014-08-01 15:13:45,994 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [44af4596] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 15:13:45,999 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [44af4596] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 15:13:46,018 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 15:13:52,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 15:13:52,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 15:13:52,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ad314e6
>2014-08-01 15:13:52,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling started: taskId = 984f683a-d0fa-4c7b-9498-505542b322f1
>2014-08-01 15:13:53,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended: taskId = 984f683a-d0fa-4c7b-9498-505542b322f1 task status = finished
>2014-08-01 15:13:53,262 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended, spm status: SPM
>2014-08-01 15:13:53,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=984f683a-d0fa-4c7b-9498-505542b322f1), log id: 258448fb
>2014-08-01 15:13:53,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, HSMClearTaskVDSCommand, log id: 258448fb
>2014-08-01 15:13:53,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63cbbedf, log id: 2ad314e6
>2014-08-01 15:13:53,327 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 15:13:53,345 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 15:13:53,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5df37e2a
>2014-08-01 15:13:53,395 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 15:13:53,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f4d8ddc
>2014-08-01 15:13:53,402 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f4d8ddc
>2014-08-01 15:13:53,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5df37e2a
>2014-08-01 15:13:53,404 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) Discovered no tasks on Storage Pool Default
>2014-08-01 15:23:55,485 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-01 15:24:23,115 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
>2014-08-01 16:23:55,493 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized
>2014-08-01 16:24:23,147 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf.
>2014-08-01 17:23:55,499 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized
>2014-08-01 17:24:23,163 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
>2014-08-01 18:13:40,434 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 18:13:40,436 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 18:15:11,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 18:15:11,467 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 18:15:11,471 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [14ab6114] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 18:15:11,473 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [14ab6114] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 18:15:11,488 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [14ab6114] Correlation ID: 14ab6114, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 18:17:11,482 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 18:17:11,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [14ab6114] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 18:17:11,486 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 18:17:11,486 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 18:17:11,535 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 18:17:11,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 356feda2
>2014-08-01 18:17:11,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 538ce19b
>2014-08-01 18:17:11,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 18:19:11,493 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 18:19:11,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] FINISH, SpmStopVDSCommand, log id: 538ce19b
>2014-08-01 18:19:11,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 18:19:11,501 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [39225a1e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 18:19:11,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 18:19:11,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] FINISH, ResetIrsVDSCommand, log id: 356feda2
>2014-08-01 18:19:11,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 18:19:11,503 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [39225a1e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 18:19:11,522 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [668577b5] Correlation ID: 668577b5, Job ID: ca102c87-ba6a-420b-bb83-e0fef82d5504, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 18:19:11,545 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [39225a1e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 18:19:11,547 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-15) [14ab6114] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 18:19:11,549 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 18:19:11,617 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 18:19:11,643 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 18:19:11,650 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 18:19:11,656 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 18:19:11,668 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [69558113] Correlation ID: 69558113, Job ID: 0c702fcf-a0ce-4d1f-8178-de9a6ca40c0c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 18:19:11,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [69558113] Removing vds [] from the domain in maintenance cache
>2014-08-01 18:19:11,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [69558113] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 18:19:11,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 18:19:11,684 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 18:19:11,701 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 18:19:11,709 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] Removing vds [] from the domain in maintenance cache
>2014-08-01 18:19:11,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 18:19:11,711 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-15) [14ab6114] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 18:19:14,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ebb7847
>2014-08-01 18:19:14,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GetHardwareInfoVDSCommand, log id: ebb7847
>2014-08-01 18:19:14,895 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [11d24648] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 18:19:14,909 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [277b9d61] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 18:19:14,918 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 18:19:14,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18d74e3
>2014-08-01 18:19:14,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 18d74e3
>2014-08-01 18:19:14,994 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Host rack storage connection was succeeded 
>2014-08-01 18:19:15,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 55f1b85c
>2014-08-01 18:19:21,515 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-98) [5aa74089] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 18:19:21,519 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-98) [5aa74089] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 18:20:15,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 55f1b85c
>2014-08-01 18:20:15,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 36068ab9
>2014-08-01 18:20:15,531 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 36068ab9
>2014-08-01 18:20:15,538 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 18:20:15,545 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 18:20:15,566 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 18:20:21,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 18:20:21,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 18:20:21,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12d526d5
>2014-08-01 18:20:21,723 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling started: taskId = 6f93fc93-89bd-42fb-897f-2baa1071d5c8
>2014-08-01 18:20:22,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended: taskId = 6f93fc93-89bd-42fb-897f-2baa1071d5c8 task status = finished
>2014-08-01 18:20:22,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended, spm status: SPM
>2014-08-01 18:20:22,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6f93fc93-89bd-42fb-897f-2baa1071d5c8), log id: 72b55b14
>2014-08-01 18:20:22,832 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, HSMClearTaskVDSCommand, log id: 72b55b14
>2014-08-01 18:20:22,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6024726b, log id: 12d526d5
>2014-08-01 18:20:22,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 18:20:22,845 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 18:20:22,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 465d34c5
>2014-08-01 18:20:22,886 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 18:20:22,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1c9b88a6
>2014-08-01 18:20:22,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c9b88a6
>2014-08-01 18:20:22,894 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 465d34c5
>2014-08-01 18:20:22,896 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) Discovered no tasks on Storage Pool Default
>2014-08-01 18:23:55,506 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized
>2014-08-01 18:24:23,180 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf.
>2014-08-01 19:23:55,512 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Principal admin::internal synchronized
>2014-08-01 19:24:23,211 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf.
>2014-08-01 20:23:55,516 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized
>2014-08-01 20:24:23,226 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
>2014-08-01 21:20:12,939 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-01 21:20:28,063 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-01 21:21:45,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 21:21:45,682 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 21:21:45,687 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [81f884c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-01 21:21:45,691 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [81f884c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-01 21:21:45,704 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [81f884c] Correlation ID: 81f884c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-01 21:23:45,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 21:23:45,752 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 21:23:45,754 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-01 21:23:45,755 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-01 21:23:45,791 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 21:23:45,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: e14837a
>2014-08-01 21:23:45,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 653e7fef
>2014-08-01 21:23:45,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-01 21:23:55,520 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) [52d379dc] Principal admin::internal synchronized
>2014-08-01 21:24:23,240 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-01 21:25:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Autorecovering 1 storage domains
>2014-08-01 21:25:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-01 21:25:00,012 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-01 21:25:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 9:25 PM
>2014-08-01 21:25:00,024 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-01 21:25:00,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38b92ae0
>2014-08-01 21:25:45,713 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-01 21:25:45,715 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] FINISH, SpmStopVDSCommand, log id: 653e7fef
>2014-08-01 21:25:45,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-01 21:25:45,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 21:25:45,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-01 21:25:45,725 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-01 21:25:45,730 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] FINISH, ResetIrsVDSCommand, log id: e14837a
>2014-08-01 21:25:45,724 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-01 21:25:45,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] FINISH, ConnectStorageServerVDSCommand, log id: 38b92ae0
>2014-08-01 21:25:45,733 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-01 21:25:45,730 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-01 21:25:45,746 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] Correlation ID: 5f8d1a83, Job ID: e1c97138-8cf1-4fb6-8a9d-5e9e4910f98d, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-01 21:25:45,737 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-01 21:25:45,767 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 9:25 PM
>2014-08-01 21:25:45,771 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [81f884c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 21:25:45,773 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-01 21:25:45,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-01 21:25:45,825 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-01 21:25:45,843 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6edda802] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-01 21:25:45,850 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6edda802] Removing vds [] from the domain in maintenance cache
>2014-08-01 21:25:45,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6edda802] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 21:25:45,852 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [81f884c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-01 21:25:45,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-01 21:25:45,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-01 21:25:45,872 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 21:25:45,878 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-01 21:25:45,883 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-01 21:25:45,892 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Correlation ID: 41751a9b, Job ID: cd8ae9ed-6d97-4de3-a4a9-6c1a40360401, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-01 21:25:45,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Removing vds [] from the domain in maintenance cache
>2014-08-01 21:25:45,897 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Removing host(s) [] from hosts unseen domain report cache
>2014-08-01 21:25:49,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5aba8df1
>2014-08-01 21:25:49,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 5aba8df1
>2014-08-01 21:25:49,125 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [158bf392] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 21:25:49,138 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [4e7820e7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 21:25:49,145 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-01 21:25:49,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5208656
>2014-08-01 21:25:49,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5208656
>2014-08-01 21:25:49,214 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] Host rack storage connection was succeeded 
>2014-08-01 21:25:49,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 76fa79
>2014-08-01 21:25:55,736 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [797bf730] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 21:25:55,740 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [797bf730] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-01 21:26:49,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, ConnectStoragePoolVDSCommand, log id: 76fa79
>2014-08-01 21:26:49,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a550198
>2014-08-01 21:26:49,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a550198
>2014-08-01 21:26:49,721 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3721c63a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-01 21:26:49,728 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3721c63a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-01 21:26:49,755 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [64566e87] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-01 21:26:55,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-01 21:26:55,873 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-01 21:26:55,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e71842e
>2014-08-01 21:26:55,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling started: taskId = c08e9839-c965-4fab-97c4-08aad7ae46a0
>2014-08-01 21:26:56,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling ended: taskId = c08e9839-c965-4fab-97c4-08aad7ae46a0 task status = finished
>2014-08-01 21:26:56,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling ended, spm status: SPM
>2014-08-01 21:26:56,989 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c08e9839-c965-4fab-97c4-08aad7ae46a0), log id: 21087e37
>2014-08-01 21:26:57,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] FINISH, HSMClearTaskVDSCommand, log id: 21087e37
>2014-08-01 21:26:57,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1358db5d, log id: 5e71842e
>2014-08-01 21:26:57,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-01 21:26:57,046 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [39225a1e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-01 21:26:57,048 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-14) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-01 21:26:57,062 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 46968ee2
>2014-08-01 21:26:57,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-01 21:26:57,087 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6e39348c
>2014-08-01 21:26:57,092 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e39348c
>2014-08-01 21:26:57,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46968ee2
>2014-08-01 21:26:57,094 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-14) [39225a1e] Discovered no tasks on Storage Pool Default
>2014-08-01 22:23:55,526 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized
>2014-08-01 22:24:23,243 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf.
>2014-08-01 23:23:55,532 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-08-01 23:24:23,273 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Attempting to update VMs/Templates Ovf.
>2014-08-02 00:23:55,538 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized
>2014-08-02 00:24:23,290 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf.
>2014-08-02 00:26:46,585 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 00:27:01,710 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 00:28:17,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 00:28:17,832 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 00:28:17,835 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 00:28:17,837 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 00:28:17,850 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Correlation ID: 6e437c9b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 00:30:17,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 00:30:17,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [6e437c9b] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 00:30:17,881 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 00:30:17,882 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 00:30:17,931 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 00:30:17,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2d74bbdd
>2014-08-02 00:30:17,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 680f5ec5
>2014-08-02 00:30:17,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 00:32:17,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 00:32:17,857 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 00:32:17,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 00:32:17,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 00:33:17,989 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-08-02 00:33:17,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] FINISH, SpmStopVDSCommand, log id: 680f5ec5
>2014-08-02 00:33:17,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] FINISH, ResetIrsVDSCommand, log id: 2d74bbdd
>2014-08-02 00:33:18,007 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] Correlation ID: 3aa22b25, Job ID: 5cc7a651-da12-49a9-ac41-77cca53d54ba, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 00:33:18,015 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6e437c9b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 00:33:18,017 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 00:33:18,073 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 00:33:18,128 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-17) [b275d22] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 00:33:18,158 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [b275d22] Removing vds [] from the domain in maintenance cache
>2014-08-02 00:33:18,159 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [b275d22] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 00:33:18,160 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6e437c9b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 00:33:18,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 00:33:18,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 00:33:18,182 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 00:33:18,188 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 00:33:18,195 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 00:33:18,205 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [688ba99] Correlation ID: 688ba99, Job ID: 7231f96a-616b-4d52-8c57-710db6b5a94b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 00:33:18,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [688ba99] Removing vds [] from the domain in maintenance cache
>2014-08-02 00:33:18,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [688ba99] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 00:34:17,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 00:34:17,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 00:34:18,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-02 00:34:18,753 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-08-02 00:34:18,766 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-02 00:34:18,775 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-02 00:34:22,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 83765ef
>2014-08-02 00:34:22,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] FINISH, GetHardwareInfoVDSCommand, log id: 83765ef
>2014-08-02 00:34:22,093 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [632bb53a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 00:34:22,142 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [632bb53a] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-02 00:34:22,144 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [65662402] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 00:34:22,152 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 00:34:22,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6a0f7872
>2014-08-02 00:34:22,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6a0f7872
>2014-08-02 00:34:22,227 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Host rack storage connection was succeeded 
>2014-08-02 00:34:22,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3ea3093e
>2014-08-02 00:34:27,874 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 00:34:27,878 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 00:35:22,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 3ea3093e
>2014-08-02 00:35:22,640 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5250ec7d
>2014-08-02 00:35:22,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5250ec7d
>2014-08-02 00:35:22,741 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 00:35:22,747 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 00:35:22,763 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [6169221c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 00:35:27,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 00:35:27,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 00:35:27,996 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d4d1318
>2014-08-02 00:35:28,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = be9a2371-7b04-4133-a3e1-599eda518827
>2014-08-02 00:35:29,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = be9a2371-7b04-4133-a3e1-599eda518827 task status = finished
>2014-08-02 00:35:29,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM
>2014-08-02 00:35:29,111 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=be9a2371-7b04-4133-a3e1-599eda518827), log id: 218f54dc
>2014-08-02 00:35:29,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 218f54dc
>2014-08-02 00:35:29,158 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@206ea66a, log id: 5d4d1318
>2014-08-02 00:35:29,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 00:35:29,168 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 00:35:29,169 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-24) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 00:35:29,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5680d7b9
>2014-08-02 00:35:29,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 00:35:29,222 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 58a1d10
>2014-08-02 00:35:29,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 58a1d10
>2014-08-02 00:35:29,229 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5680d7b9
>2014-08-02 00:35:29,230 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-24) Discovered no tasks on Storage Pool Default
>2014-08-02 00:56:36,377 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-39) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 01:23:55,542 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Principal admin::internal synchronized
>2014-08-02 01:24:23,306 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf.
>2014-08-02 02:23:55,551 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized
>2014-08-02 02:24:23,324 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-08-02 03:23:55,560 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) [56f1581c] Principal admin::internal synchronized
>2014-08-02 03:24:23,356 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf.
>2014-08-02 03:35:08,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 03:35:23,623 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Start deleteAgedOutAuditLogs
>2014-08-02 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Start deleteAgedOutCommandEntities
>2014-08-02 03:35:35,007 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Finished deleteAgedOutAuditLogs
>2014-08-02 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Finished deleteAgedOutCommandEntities
>2014-08-02 03:36:48,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 03:36:48,105 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 03:36:48,108 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-86) [6452f878] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 03:36:48,110 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-86) [6452f878] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 03:36:48,125 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [6452f878] Correlation ID: 6452f878, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 03:38:48,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 03:38:48,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [6452f878] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 03:38:48,168 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 03:38:48,169 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 03:38:48,209 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 03:38:48,230 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6aa9a5ad
>2014-08-02 03:38:48,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 79d6077b
>2014-08-02 03:38:48,245 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 03:40:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering 1 storage domains
>2014-08-02 03:40:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-02 03:40:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 03:40:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 3:40 AM
>2014-08-02 03:40:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-02 03:40:00,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 56ca28f
>2014-08-02 03:40:48,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 03:40:48,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 03:40:48,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 03:40:48,156 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 03:41:48,151 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 03:41:48,155 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] FINISH, SpmStopVDSCommand, log id: 79d6077b
>2014-08-02 03:41:48,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 03:41:48,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 03:41:48,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 03:41:48,167 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) [81f884c] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 03:41:48,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] FINISH, ConnectStorageServerVDSCommand, log id: 56ca28f
>2014-08-02 03:41:48,173 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-02 03:41:48,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] FINISH, ResetIrsVDSCommand, log id: 6aa9a5ad
>2014-08-02 03:41:48,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 03:41:48,171 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) [81f884c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 03:41:48,190 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [31614d3b] Correlation ID: 31614d3b, Job ID: 126a434b-1c98-4772-9e8d-1e36173331fd, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 03:41:48,182 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-02 03:41:48,213 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 3:41 AM
>2014-08-02 03:41:48,219 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) [81f884c] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 03:41:48,221 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-24) [6452f878] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 03:41:48,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 03:41:48,285 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 03:41:48,303 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [48543fa4] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 03:41:48,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [48543fa4] Removing vds [] from the domain in maintenance cache
>2014-08-02 03:41:48,311 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [48543fa4] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 03:41:48,312 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-24) [6452f878] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 03:41:48,314 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 03:41:48,319 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 03:41:48,333 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 03:41:48,340 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 03:41:48,346 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 03:41:48,356 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [159b9156] Correlation ID: 159b9156, Job ID: ab9af258-7dea-45e2-8fae-f85d922dfae0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 03:41:48,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [159b9156] Removing vds [] from the domain in maintenance cache
>2014-08-02 03:41:48,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [159b9156] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 03:41:51,470 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ac91fab
>2014-08-02 03:41:51,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] FINISH, GetHardwareInfoVDSCommand, log id: ac91fab
>2014-08-02 03:41:51,551 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [50676e63] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 03:41:51,600 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-82) [5f8d8487] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 03:41:51,608 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 03:41:51,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 45b24c41
>2014-08-02 03:41:51,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 45b24c41
>2014-08-02 03:41:51,679 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] Host rack storage connection was succeeded 
>2014-08-02 03:41:51,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5acd2e98
>2014-08-02 03:41:58,184 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-83) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 03:41:58,188 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-83) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 03:42:52,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 5acd2e98
>2014-08-02 03:42:52,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4ed6dec
>2014-08-02 03:42:52,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4ed6dec
>2014-08-02 03:42:52,198 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [f27a43f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 03:42:52,205 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [f27a43f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 03:42:52,224 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 03:42:58,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 03:42:58,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 03:42:58,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ee604ee
>2014-08-02 03:42:58,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling started: taskId = 428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634
>2014-08-02 03:42:59,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling ended: taskId = 428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634 task status = finished
>2014-08-02 03:42:59,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling ended, spm status: SPM
>2014-08-02 03:42:59,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634), log id: 3740354e
>2014-08-02 03:42:59,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] FINISH, HSMClearTaskVDSCommand, log id: 3740354e
>2014-08-02 03:42:59,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@885a4ff, log id: 3ee604ee
>2014-08-02 03:42:59,539 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 03:42:59,546 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [375fcc6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 03:42:59,548 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 03:42:59,568 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65eaae20
>2014-08-02 03:42:59,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 03:42:59,594 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7938381d
>2014-08-02 03:42:59,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7938381d
>2014-08-02 03:42:59,601 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 65eaae20
>2014-08-02 03:42:59,602 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) [375fcc6] Discovered no tasks on Storage Pool Default
>2014-08-02 04:23:55,565 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) [7526d482] Principal admin::internal synchronized
>2014-08-02 04:24:23,374 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
>2014-08-02 05:23:55,571 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized
>2014-08-02 05:24:23,387 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf.
>2014-08-02 06:23:55,585 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-08-02 06:24:23,401 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-02 06:26:17,467 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-18) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 06:42:40,616 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 06:42:40,618 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 06:44:13,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 06:44:13,992 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 06:44:13,995 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 06:44:13,997 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 06:44:14,011 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Correlation ID: 72c79e4e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 06:46:14,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 06:46:14,041 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 06:46:14,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 06:46:14,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 06:46:14,076 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 06:46:14,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7e61050b
>2014-08-02 06:46:14,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 10e50f4b
>2014-08-02 06:46:14,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 06:48:14,011 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 06:48:14,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] FINISH, SpmStopVDSCommand, log id: 10e50f4b
>2014-08-02 06:48:14,016 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-88) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 06:48:14,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 06:48:14,018 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 06:48:14,021 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 06:48:14,024 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] FINISH, ResetIrsVDSCommand, log id: 7e61050b
>2014-08-02 06:48:14,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 06:48:14,034 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] Correlation ID: 27ab8ac7, Job ID: 99448b18-cda2-4769-96af-892b76b11b35, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 06:48:14,053 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 06:48:14,057 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [72c79e4e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 06:48:14,059 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 06:48:14,125 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 06:48:14,141 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 06:48:14,148 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 06:48:14,154 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 06:48:14,163 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Correlation ID: 42b39b7a, Job ID: f218d5ce-cd5c-46a6-869e-f4a01de4c40d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 06:48:14,168 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Removing vds [] from the domain in maintenance cache
>2014-08-02 06:48:14,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 06:48:14,171 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 06:48:14,176 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 06:48:14,193 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-13) [403a3975] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 06:48:14,199 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) [403a3975] Removing vds [] from the domain in maintenance cache
>2014-08-02 06:48:14,200 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) [403a3975] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 06:48:14,201 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [72c79e4e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 06:48:17,303 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 602557e9
>2014-08-02 06:48:17,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] FINISH, GetHardwareInfoVDSCommand, log id: 602557e9
>2014-08-02 06:48:17,383 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [1fef041f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 06:48:17,397 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [4ae34136] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 06:48:17,405 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 06:48:17,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d22a875
>2014-08-02 06:48:17,477 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: d22a875
>2014-08-02 06:48:17,479 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] Host rack storage connection was succeeded 
>2014-08-02 06:48:17,486 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 22740ece
>2014-08-02 06:48:24,027 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 06:48:24,031 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 06:49:17,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 22740ece
>2014-08-02 06:49:17,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 18e75966
>2014-08-02 06:49:18,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] FINISH, SetMOMPolicyParametersVDSCommand, log id: 18e75966
>2014-08-02 06:49:18,054 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [39b61b80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 06:49:18,061 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [39b61b80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 06:49:18,083 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [778da158] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 06:49:24,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 06:49:24,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 06:49:24,217 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4afb8e4a
>2014-08-02 06:49:24,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling started: taskId = 48f1f0a3-dfd7-41e7-a98c-0648e13a68f4
>2014-08-02 06:49:25,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling ended: taskId = 48f1f0a3-dfd7-41e7-a98c-0648e13a68f4 task status = finished
>2014-08-02 06:49:25,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling ended, spm status: SPM
>2014-08-02 06:49:25,330 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=48f1f0a3-dfd7-41e7-a98c-0648e13a68f4), log id: 49d938e4
>2014-08-02 06:49:25,376 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] FINISH, HSMClearTaskVDSCommand, log id: 49d938e4
>2014-08-02 06:49:25,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f60e75e, log id: 4afb8e4a
>2014-08-02 06:49:25,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 06:49:25,387 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 06:49:25,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2152aa5f
>2014-08-02 06:49:25,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 06:49:25,435 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3b43c857
>2014-08-02 06:49:25,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b43c857
>2014-08-02 06:49:25,442 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2152aa5f
>2014-08-02 06:49:25,443 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] Discovered no tasks on Storage Pool Default
>2014-08-02 07:04:10,278 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 07:23:55,592 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) [42c2440e] Principal admin::internal synchronized
>2014-08-02 07:24:23,414 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf.
>2014-08-02 08:23:55,597 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [2e3238db] Principal admin::internal synchronized
>2014-08-02 08:24:23,427 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf.
>2014-08-02 08:27:38,636 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 09:23:55,603 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2809936b] Principal admin::internal synchronized
>2014-08-02 09:24:23,443 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
>2014-08-02 09:49:00,318 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 09:49:15,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-08-02 09:49:15,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-02 09:54:02,727 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 09:54:02,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 09:55:41,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [52a75f18] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 09:55:41,517 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 09:55:41,521 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 09:55:41,524 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 09:55:41,543 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Correlation ID: 3b9a9fa6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 09:57:41,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 09:57:41,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 09:57:41,561 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 09:57:41,562 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 09:57:41,596 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 09:57:41,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5bf4ad33
>2014-08-02 09:57:41,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3b7ebd93
>2014-08-02 09:57:41,622 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 09:59:41,534 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 09:59:41,538 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] FINISH, SpmStopVDSCommand, log id: 3b7ebd93
>2014-08-02 09:59:41,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-92) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 09:59:41,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 09:59:41,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [6452f878] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 09:59:41,541 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 09:59:41,545 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] FINISH, ResetIrsVDSCommand, log id: 5bf4ad33
>2014-08-02 09:59:41,546 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 09:59:41,555 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [1e342011] Correlation ID: 1e342011, Job ID: d536ca2f-e474-44f5-85fd-b25232ac2e26, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 09:59:41,584 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 09:59:41,585 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [3b9a9fa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 09:59:41,587 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 09:59:41,652 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 09:59:41,683 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 09:59:41,691 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] Removing vds [] from the domain in maintenance cache
>2014-08-02 09:59:41,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 09:59:41,693 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [3b9a9fa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 09:59:41,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 09:59:41,700 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 09:59:41,715 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 09:59:41,722 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 09:59:41,728 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 09:59:41,737 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Correlation ID: 3fa1a8be, Job ID: 951aa6d2-965b-4f36-9026-c6075f7dffcb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 09:59:41,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Removing vds [] from the domain in maintenance cache
>2014-08-02 09:59:41,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 09:59:44,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 240a17da
>2014-08-02 09:59:44,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 240a17da
>2014-08-02 09:59:44,923 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [37653756] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 09:59:44,977 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [5c87d5a2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 09:59:44,985 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 09:59:45,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 31af56e4
>2014-08-02 09:59:45,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 31af56e4
>2014-08-02 09:59:45,028 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Host rack storage connection was succeeded 
>2014-08-02 09:59:45,036 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4e3184fd
>2014-08-02 09:59:51,548 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 09:59:51,554 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 10:00:45,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 4e3184fd
>2014-08-02 10:00:45,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5f639bd9
>2014-08-02 10:00:45,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5f639bd9
>2014-08-02 10:00:45,560 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 10:00:45,566 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 10:00:45,607 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [7125e62c] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 10:00:51,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 10:00:51,718 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 10:00:51,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68224b
>2014-08-02 10:00:51,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling started: taskId = 477dd556-7eca-44ea-a489-060fa406db93
>2014-08-02 10:00:52,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended: taskId = 477dd556-7eca-44ea-a489-060fa406db93 task status = finished
>2014-08-02 10:00:52,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended, spm status: SPM
>2014-08-02 10:00:52,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=477dd556-7eca-44ea-a489-060fa406db93), log id: 85e6f69
>2014-08-02 10:00:52,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: 85e6f69
>2014-08-02 10:00:52,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5fcad40a, log id: 68224b
>2014-08-02 10:00:52,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 10:00:52,896 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 10:00:52,898 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-26) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 10:00:52,908 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2b69f398
>2014-08-02 10:00:52,941 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 10:00:52,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 56712782
>2014-08-02 10:00:52,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56712782
>2014-08-02 10:00:52,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b69f398
>2014-08-02 10:00:52,953 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) Discovered no tasks on Storage Pool Default
>2014-08-02 10:10:59,639 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 10:23:55,610 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-08-02 10:24:23,457 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf.
>2014-08-02 11:23:55,620 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-02 11:24:23,491 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Attempting to update VMs/Templates Ovf.
>2014-08-02 12:23:55,630 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized
>2014-08-02 12:24:23,504 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [7526d482] Attempting to update VMs/Templates Ovf.
>2014-08-02 13:00:37,098 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 13:00:37,100 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 13:02:08,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [6169221c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 13:02:08,324 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 13:02:08,326 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [3cf3236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 13:02:08,328 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [3cf3236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 13:02:08,340 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3cf3236] Correlation ID: 3cf3236, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 13:04:08,372 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 13:04:08,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3cf3236] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 13:04:08,375 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 13:04:08,376 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 13:04:08,404 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 13:04:08,421 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6e5dd12f
>2014-08-02 13:04:08,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 400c23a6
>2014-08-02 13:04:08,430 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 13:05:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering 1 storage domains
>2014-08-02 13:05:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-02 13:05:00,006 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 13:05:00,007 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 1:05 PM
>2014-08-02 13:05:00,015 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-02 13:05:00,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12c5b7a9
>2014-08-02 13:06:08,346 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 13:06:08,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] FINISH, SpmStopVDSCommand, log id: 400c23a6
>2014-08-02 13:06:08,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 13:06:08,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 13:06:08,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 13:06:08,362 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 13:06:08,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] FINISH, ConnectStorageServerVDSCommand, log id: 12c5b7a9
>2014-08-02 13:06:08,365 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 13:06:08,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] FINISH, ResetIrsVDSCommand, log id: 6e5dd12f
>2014-08-02 13:06:08,367 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-02 13:06:08,368 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 13:06:08,388 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [58dd099e] Correlation ID: 58dd099e, Job ID: 77c660cf-ebdc-4b4d-8ba8-d36a4d23524a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 13:06:08,373 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-02 13:06:08,400 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 1:06 PM
>2014-08-02 13:06:08,402 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-20) [3cf3236] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 13:06:08,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 13:06:08,408 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 13:06:08,458 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 13:06:08,478 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 13:06:08,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] Removing vds [] from the domain in maintenance cache
>2014-08-02 13:06:08,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 13:06:08,487 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-20) [3cf3236] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 13:06:08,489 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 13:06:08,493 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 13:06:08,507 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 13:06:08,514 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 13:06:08,519 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 13:06:08,530 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [7851471] Correlation ID: 7851471, Job ID: 2ff17a4f-38ca-43ac-b82d-f386c591489e, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 13:06:08,535 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [7851471] Removing vds [] from the domain in maintenance cache
>2014-08-02 13:06:08,536 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [7851471] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 13:06:11,653 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 48714368
>2014-08-02 13:06:11,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, GetHardwareInfoVDSCommand, log id: 48714368
>2014-08-02 13:06:11,729 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [46213537] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 13:06:11,742 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [239799d0] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 13:06:11,749 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 13:06:11,755 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 41607b
>2014-08-02 13:06:11,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 41607b
>2014-08-02 13:06:11,818 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Host rack storage connection was succeeded 
>2014-08-02 13:06:11,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 117dd06e
>2014-08-02 13:06:18,370 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-10) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 13:06:18,374 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-10) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 13:07:12,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 117dd06e
>2014-08-02 13:07:12,240 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2a022410
>2014-08-02 13:07:12,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2a022410
>2014-08-02 13:07:12,341 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 13:07:12,352 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 13:07:12,372 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [14cac44e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 13:07:18,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 13:07:18,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 13:07:18,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2384de27
>2014-08-02 13:07:18,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling started: taskId = eea8bd7b-857f-40af-92f5-21f814dc073a
>2014-08-02 13:07:19,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended: taskId = eea8bd7b-857f-40af-92f5-21f814dc073a task status = finished
>2014-08-02 13:07:19,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended, spm status: SPM
>2014-08-02 13:07:19,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=eea8bd7b-857f-40af-92f5-21f814dc073a), log id: 65abbc00
>2014-08-02 13:07:19,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: 65abbc00
>2014-08-02 13:07:19,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@559e77b1, log id: 2384de27
>2014-08-02 13:07:19,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 13:07:19,676 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 13:07:19,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27d5af23
>2014-08-02 13:07:19,721 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 13:07:19,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4122154e
>2014-08-02 13:07:19,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4122154e
>2014-08-02 13:07:19,730 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27d5af23
>2014-08-02 13:07:19,731 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) Discovered no tasks on Storage Pool Default
>2014-08-02 13:23:55,641 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized
>2014-08-02 13:24:23,520 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5497a121] Attempting to update VMs/Templates Ovf.
>2014-08-02 14:23:55,648 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) [64566e87] Principal admin::internal synchronized
>2014-08-02 14:24:23,555 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) [14ab6114] Attempting to update VMs/Templates Ovf.
>2014-08-02 15:23:55,655 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized
>2014-08-02 15:24:23,566 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf.
>2014-08-02 16:06:55,210 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 16:07:10,328 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-08-02 16:07:10,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-02 16:12:12,933 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 16:12:12,935 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 16:13:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 16:13:38,854 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 16:13:38,857 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 16:13:38,859 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 16:13:38,870 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Correlation ID: 5b4b3aec, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 16:15:38,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 16:15:38,877 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 16:15:38,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 16:15:38,879 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 16:15:38,918 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 16:15:38,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7fc31ce5
>2014-08-02 16:15:38,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 25df98b5
>2014-08-02 16:15:38,946 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 16:17:38,848 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 16:17:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 16:17:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 16:17:38,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 16:17:38,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] FINISH, SpmStopVDSCommand, log id: 25df98b5
>2014-08-02 16:17:38,855 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 16:17:38,858 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 16:17:38,867 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] FINISH, ResetIrsVDSCommand, log id: 7fc31ce5
>2014-08-02 16:17:38,891 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 16:17:38,896 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] Correlation ID: 69a4c1a7, Job ID: becb973f-0f07-4edb-a6b4-483fa838249a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 16:17:38,902 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [5b4b3aec] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 16:17:38,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 16:17:38,956 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 16:17:38,982 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [17131d67] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 16:17:38,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [17131d67] Removing vds [] from the domain in maintenance cache
>2014-08-02 16:17:38,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [17131d67] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 16:17:38,991 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [5b4b3aec] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 16:17:38,993 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 16:17:38,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 16:17:39,011 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 16:17:39,017 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 16:17:39,023 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 16:17:39,031 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [d22051a] Correlation ID: d22051a, Job ID: 43c67060-04e2-4496-bffc-59a9c70140cf, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 16:17:39,037 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [d22051a] Removing vds [] from the domain in maintenance cache
>2014-08-02 16:17:39,038 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [d22051a] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 16:17:42,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 435b5802
>2014-08-02 16:17:42,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GetHardwareInfoVDSCommand, log id: 435b5802
>2014-08-02 16:17:42,231 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [5d34f3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 16:17:42,241 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-70) [7b012582] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 16:17:42,251 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 16:17:42,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 420fb9ad
>2014-08-02 16:17:42,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 420fb9ad
>2014-08-02 16:17:42,317 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] Host rack storage connection was succeeded 
>2014-08-02 16:17:42,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 51df7ecf
>2014-08-02 16:17:48,860 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 16:17:48,866 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 16:18:42,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, ConnectStoragePoolVDSCommand, log id: 51df7ecf
>2014-08-02 16:18:42,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1baf9cdb
>2014-08-02 16:18:42,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1baf9cdb
>2014-08-02 16:18:42,828 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7131f834] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 16:18:42,833 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7131f834] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 16:18:42,849 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [593e74a7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 16:18:48,985 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 16:18:48,989 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 16:18:48,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65f4d717
>2014-08-02 16:18:49,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = eded74c1-eabd-4e84-a1c6-43f09092abb4
>2014-08-02 16:18:50,046 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = eded74c1-eabd-4e84-a1c6-43f09092abb4 task status = finished
>2014-08-02 16:18:50,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM
>2014-08-02 16:18:50,103 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=eded74c1-eabd-4e84-a1c6-43f09092abb4), log id: 4b4ea4c3
>2014-08-02 16:18:50,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 4b4ea4c3
>2014-08-02 16:18:50,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59095303, log id: 65f4d717
>2014-08-02 16:18:50,153 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 16:18:50,159 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 16:18:50,167 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 23fc4bab
>2014-08-02 16:18:50,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 16:18:50,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 190252c0
>2014-08-02 16:18:50,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 190252c0
>2014-08-02 16:18:50,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23fc4bab
>2014-08-02 16:18:50,214 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) Discovered no tasks on Storage Pool Default
>2014-08-02 16:23:55,663 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized
>2014-08-02 16:24:23,581 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf.
>2014-08-02 17:23:55,673 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Principal admin::internal synchronized
>2014-08-02 17:24:23,600 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf.
>2014-08-02 18:23:55,682 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized
>2014-08-02 18:24:23,639 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-08-02 19:18:30,981 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 19:18:30,983 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 19:20:05,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 19:20:05,876 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 19:20:05,879 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [22578e65] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 19:20:05,881 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [22578e65] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 19:20:05,892 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [22578e65] Correlation ID: 22578e65, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 19:22:05,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 19:22:05,946 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [22578e65] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 19:22:05,947 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 19:22:05,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 19:22:05,987 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 19:22:06,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 35957991
>2014-08-02 19:22:06,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 32514355
>2014-08-02 19:22:06,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 19:23:55,688 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized
>2014-08-02 19:24:05,915 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 19:24:05,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] FINISH, SpmStopVDSCommand, log id: 32514355
>2014-08-02 19:24:05,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 19:24:05,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 19:24:05,930 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 19:24:05,928 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 19:24:05,928 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] FINISH, ResetIrsVDSCommand, log id: 35957991
>2014-08-02 19:24:05,933 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [1b0a6877] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 19:24:05,935 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 19:24:05,948 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] Correlation ID: 3dec9b1a, Job ID: a4178e58-b69f-40cd-be52-5765c6b35483, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 19:24:05,976 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [22578e65] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 19:24:05,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 19:24:05,977 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 19:24:06,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 19:24:06,048 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 19:24:06,054 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 19:24:06,060 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 19:24:06,068 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Correlation ID: a7c3897, Job ID: 223dabc7-87ac-4af6-a6d3-a1982e8b6fcb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 19:24:06,074 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Removing vds [] from the domain in maintenance cache
>2014-08-02 19:24:06,075 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 19:24:06,077 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 19:24:06,081 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 19:24:06,102 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [8828708] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 19:24:06,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [8828708] Removing vds [] from the domain in maintenance cache
>2014-08-02 19:24:06,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [8828708] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 19:24:06,111 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [22578e65] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 19:24:09,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 38c6a84c
>2014-08-02 19:24:09,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] FINISH, GetHardwareInfoVDSCommand, log id: 38c6a84c
>2014-08-02 19:24:09,310 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [28763d45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 19:24:09,360 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [f3cb3e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 19:24:09,368 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 19:24:09,381 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 24fbafd6
>2014-08-02 19:24:09,442 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 24fbafd6
>2014-08-02 19:24:09,443 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] Host rack storage connection was succeeded 
>2014-08-02 19:24:09,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3889c774
>2014-08-02 19:24:15,935 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 19:24:15,939 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 19:24:23,653 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Attempting to update VMs/Templates Ovf.
>2014-08-02 19:25:09,900 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, ConnectStoragePoolVDSCommand, log id: 3889c774
>2014-08-02 19:25:09,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d5119a3
>2014-08-02 19:25:10,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d5119a3
>2014-08-02 19:25:10,014 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2bee177c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 19:25:10,019 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2bee177c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 19:25:10,043 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [1802a60f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 19:25:16,047 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 19:25:16,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 19:25:16,053 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 394c868
>2014-08-02 19:25:16,101 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling started: taskId = 86badb6f-afcd-4701-bfc8-aed1cd304126
>2014-08-02 19:25:17,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling ended: taskId = 86badb6f-afcd-4701-bfc8-aed1cd304126 task status = finished
>2014-08-02 19:25:17,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling ended, spm status: SPM
>2014-08-02 19:25:17,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=86badb6f-afcd-4701-bfc8-aed1cd304126), log id: 279944b4
>2014-08-02 19:25:17,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, HSMClearTaskVDSCommand, log id: 279944b4
>2014-08-02 19:25:17,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bcb2b28, log id: 394c868
>2014-08-02 19:25:17,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 19:25:17,222 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 19:25:17,224 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-02 19:25:17,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 40042922
>2014-08-02 19:25:17,261 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 19:25:17,263 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 163b9d04
>2014-08-02 19:25:17,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 163b9d04
>2014-08-02 19:25:17,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40042922
>2014-08-02 19:25:17,270 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default
>2014-08-02 20:23:55,694 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Principal admin::internal synchronized
>2014-08-02 20:24:23,656 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf.
>2014-08-02 21:23:55,699 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized
>2014-08-02 21:24:23,670 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf.
>2014-08-02 22:23:55,707 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized
>2014-08-02 22:24:23,693 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
>2014-08-02 22:25:04,483 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-02 22:25:19,603 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-02 22:26:43,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 22:26:43,855 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 22:26:43,859 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [16080eb6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-02 22:26:43,861 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [16080eb6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-02 22:26:43,875 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [16080eb6] Correlation ID: 16080eb6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-02 22:28:43,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 22:28:43,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 22:28:43,903 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-02 22:28:43,904 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-02 22:28:43,949 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 22:28:43,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 63d0b4df
>2014-08-02 22:28:43,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3e2ae127
>2014-08-02 22:28:43,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-02 22:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) Autorecovering 1 storage domains
>2014-08-02 22:30:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-02 22:30:00,007 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-02 22:30:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 10:30 PM
>2014-08-02 22:30:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-02 22:30:00,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6544c2ce
>2014-08-02 22:30:43,871 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-02 22:30:43,874 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] FINISH, SpmStopVDSCommand, log id: 3e2ae127
>2014-08-02 22:30:43,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 22:30:43,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-02 22:30:43,880 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-02 22:30:43,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] FINISH, ConnectStorageServerVDSCommand, log id: 6544c2ce
>2014-08-02 22:30:43,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-02 22:30:43,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-02 22:30:43,884 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-02 22:30:43,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-02 22:30:43,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] FINISH, ResetIrsVDSCommand, log id: 63d0b4df
>2014-08-02 22:30:43,896 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-02 22:30:43,914 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 10:30 PM
>2014-08-02 22:30:43,920 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-02 22:30:43,927 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] Correlation ID: 4be6f2c6, Job ID: 181aefb7-a979-458f-8d7f-96e080fe365b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-02 22:30:43,935 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [16080eb6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 22:30:43,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-02 22:30:44,020 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-02 22:30:44,062 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [398c64db] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-02 22:30:44,070 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [398c64db] Removing vds [] from the domain in maintenance cache
>2014-08-02 22:30:44,071 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [398c64db] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 22:30:44,072 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [16080eb6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-02 22:30:44,074 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-02 22:30:44,079 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-02 22:30:44,095 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 22:30:44,102 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-02 22:30:44,107 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-02 22:30:44,131 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [2243644b] Correlation ID: 2243644b, Job ID: 62ef5044-ff5a-4bad-b2fc-61d7dbfb3fd2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-02 22:30:44,138 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [2243644b] Removing vds [] from the domain in maintenance cache
>2014-08-02 22:30:44,140 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [2243644b] Removing host(s) [] from hosts unseen domain report cache
>2014-08-02 22:30:47,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4c2dc6
>2014-08-02 22:30:47,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] FINISH, GetHardwareInfoVDSCommand, log id: 4c2dc6
>2014-08-02 22:30:47,275 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [6eca077a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 22:30:47,324 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [2bbfe85] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 22:30:47,332 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-02 22:30:47,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6ded47ec
>2014-08-02 22:30:47,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6ded47ec
>2014-08-02 22:30:47,405 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] Host rack storage connection was succeeded 
>2014-08-02 22:30:47,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: d791798
>2014-08-02 22:30:53,888 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 22:30:53,892 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-02 22:31:47,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: d791798
>2014-08-02 22:31:47,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7b057c2c
>2014-08-02 22:31:47,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7b057c2c
>2014-08-02 22:31:47,933 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [414b367e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-02 22:31:47,942 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [414b367e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-02 22:31:47,962 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-02 22:31:54,051 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-02 22:31:54,060 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-02 22:31:54,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d4de280
>2014-08-02 22:31:54,118 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling started: taskId = 6ba464a7-bab7-470e-941b-7eb78762f619
>2014-08-02 22:31:55,124 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended: taskId = 6ba464a7-bab7-470e-941b-7eb78762f619 task status = finished
>2014-08-02 22:31:55,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended, spm status: SPM
>2014-08-02 22:31:55,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6ba464a7-bab7-470e-941b-7eb78762f619), log id: 10bf3e13
>2014-08-02 22:31:55,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, HSMClearTaskVDSCommand, log id: 10bf3e13
>2014-08-02 22:31:55,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f666b43, log id: 5d4de280
>2014-08-02 22:31:55,233 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-02 22:31:55,239 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-02 22:31:55,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f1c5ade
>2014-08-02 22:31:55,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-02 22:31:55,284 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7bfa6bec
>2014-08-02 22:31:55,290 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bfa6bec
>2014-08-02 22:31:55,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f1c5ade
>2014-08-02 22:31:55,292 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-10) Discovered no tasks on Storage Pool Default
>2014-08-02 23:23:55,713 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Principal admin::internal synchronized
>2014-08-02 23:24:23,730 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) [5e890dae] Attempting to update VMs/Templates Ovf.
>2014-08-03 00:23:55,723 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized
>2014-08-03 00:24:23,746 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf.
>2014-08-03 01:23:55,743 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-03 01:24:23,759 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Attempting to update VMs/Templates Ovf.
>2014-08-03 01:31:40,435 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 01:31:40,437 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 01:31:55,560 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-08-03 01:31:55,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-03 01:31:55,564 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export recovered from problem. vds: rack
>2014-08-03 01:31:55,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-03 01:36:27,968 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 01:36:27,969 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 01:39:17,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 01:39:17,592 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 01:39:17,596 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 01:39:17,598 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 01:39:17,612 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Correlation ID: 3c2dfb77, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 01:41:17,640 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 01:41:17,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 01:41:17,646 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 01:41:17,647 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 01:41:17,682 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 01:41:17,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 51a0fdcc
>2014-08-03 01:41:17,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3efb9be1
>2014-08-03 01:41:17,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 01:43:17,611 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 01:43:17,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] FINISH, SpmStopVDSCommand, log id: 3efb9be1
>2014-08-03 01:43:17,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 01:43:17,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 01:43:17,623 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 01:43:17,626 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [686e64a1] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 01:43:17,628 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [686e64a1] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 01:43:17,630 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] FINISH, ResetIrsVDSCommand, log id: 51a0fdcc
>2014-08-03 01:43:17,629 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [686e64a1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 01:43:17,640 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [3c268f29] Correlation ID: 3c268f29, Job ID: 12ffb774-0b76-4f7f-a006-17a9047fcc90, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 01:43:17,658 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [686e64a1] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 01:43:17,659 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [3c2dfb77] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 01:43:17,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 01:43:17,730 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 01:43:17,746 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 01:43:17,753 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 01:43:17,759 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 01:43:17,770 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Correlation ID: 78f1ddd1, Job ID: 26bd13d4-17ac-45b5-a9c8-a297362db7f8, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 01:43:17,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Removing vds [] from the domain in maintenance cache
>2014-08-03 01:43:17,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 01:43:17,777 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 01:43:17,782 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 01:43:17,800 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 01:43:17,807 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] Removing vds [] from the domain in maintenance cache
>2014-08-03 01:43:17,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 01:43:17,809 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [3c2dfb77] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 01:43:21,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 57cdbcee
>2014-08-03 01:43:21,163 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 57cdbcee
>2014-08-03 01:43:21,188 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [1f9b3de5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 01:43:21,238 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [1d78fb26] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 01:43:21,246 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 01:43:21,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6201d59d
>2014-08-03 01:43:21,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6201d59d
>2014-08-03 01:43:21,321 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] Host rack storage connection was succeeded 
>2014-08-03 01:43:21,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1a26ec0c
>2014-08-03 01:43:27,631 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 01:43:27,634 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 01:44:21,726 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 1a26ec0c
>2014-08-03 01:44:21,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42a37983
>2014-08-03 01:44:21,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] FINISH, SetMOMPolicyParametersVDSCommand, log id: 42a37983
>2014-08-03 01:44:21,880 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [311da2ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 01:44:21,887 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [311da2ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 01:44:21,906 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [aa1055d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 01:44:27,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 01:44:27,780 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 01:44:27,782 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70b313e7
>2014-08-03 01:44:27,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling started: taskId = af93a1cf-cc2f-4df7-8929-868095e457a5
>2014-08-03 01:44:28,837 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling ended: taskId = af93a1cf-cc2f-4df7-8929-868095e457a5 task status = finished
>2014-08-03 01:44:28,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling ended, spm status: SPM
>2014-08-03 01:44:28,895 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=af93a1cf-cc2f-4df7-8929-868095e457a5), log id: 1ef7fc4e
>2014-08-03 01:44:28,940 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] FINISH, HSMClearTaskVDSCommand, log id: 1ef7fc4e
>2014-08-03 01:44:28,941 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@51dd38bb, log id: 70b313e7
>2014-08-03 01:44:28,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 01:44:28,951 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [37510499] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 01:44:28,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3dd473ae
>2014-08-03 01:44:28,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 01:44:28,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42baa90c
>2014-08-03 01:44:29,002 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42baa90c
>2014-08-03 01:44:29,003 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dd473ae
>2014-08-03 01:44:29,004 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) [37510499] Discovered no tasks on Storage Pool Default
>2014-08-03 02:23:55,750 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Principal admin::internal synchronized
>2014-08-03 02:24:23,791 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Attempting to update VMs/Templates Ovf.
>2014-08-03 03:23:55,760 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) [64566e87] Principal admin::internal synchronized
>2014-08-03 03:24:23,827 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-08-03 03:35:35,000 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Start deleteAgedOutAuditLogs
>2014-08-03 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) [6199900c] Start deleteAgedOutCommandEntities
>2014-08-03 03:35:35,004 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Finished deleteAgedOutAuditLogs
>2014-08-03 03:35:35,007 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) [6199900c] Finished deleteAgedOutCommandEntities
>2014-08-03 04:23:55,772 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Principal admin::internal synchronized
>2014-08-03 04:24:23,860 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) [5aa74089] Attempting to update VMs/Templates Ovf.
>2014-08-03 04:39:14,569 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 04:39:14,571 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 04:39:44,844 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack
>2014-08-03 04:39:44,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-03 04:39:44,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export recovered from problem. vds: rack
>2014-08-03 04:39:44,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
>2014-08-03 04:44:17,058 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 04:44:17,060 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 04:46:52,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 04:46:52,478 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 04:46:52,482 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 04:46:52,484 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 04:46:52,495 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [8557062] Correlation ID: 8557062, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 04:48:52,526 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 04:48:52,530 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) [8557062] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 04:48:52,531 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [8557062] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 04:48:52,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 04:48:52,579 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 04:48:52,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5d22bd8
>2014-08-03 04:48:52,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 40ecc90d
>2014-08-03 04:48:52,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 04:50:00,010 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Autorecovering 1 storage domains
>2014-08-03 04:50:00,011 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-03 04:50:00,013 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 04:50:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 4:50 AM
>2014-08-03 04:50:00,025 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-03 04:50:00,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 684c00d3
>2014-08-03 04:50:52,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 04:50:52,511 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-11) [aeb0141] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 04:50:52,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 04:50:52,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 04:51:52,504 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 04:51:52,506 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] FINISH, SpmStopVDSCommand, log id: 40ecc90d
>2014-08-03 04:51:52,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 04:51:52,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 04:51:52,512 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetAllVmStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 04:51:52,519 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 04:51:52,519 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 04:51:52,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] FINISH, ConnectStorageServerVDSCommand, log id: 684c00d3
>2014-08-03 04:51:52,521 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] FINISH, ResetIrsVDSCommand, log id: 5d22bd8
>2014-08-03 04:51:52,522 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getAllVmStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 04:51:52,534 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [766b439f] Correlation ID: 766b439f, Job ID: 0daff75a-6375-4171-89ce-94b17606f321, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 04:51:52,525 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-03 04:51:52,559 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-14) [8557062] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 04:51:52,576 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-03 04:51:52,579 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 4:51 AM
>2014-08-03 04:51:52,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 04:51:52,600 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 04:51:52,672 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 04:51:52,727 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 04:51:52,735 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 04:51:52,742 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 04:51:52,755 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [472f3970] Correlation ID: 472f3970, Job ID: 9a3762fb-248b-4198-b8c0-6fcf3ea4f970, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 04:51:52,762 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [472f3970] Removing vds [] from the domain in maintenance cache
>2014-08-03 04:51:52,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [472f3970] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 04:51:52,776 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 04:51:52,782 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 04:51:52,812 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [1c85032d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 04:51:52,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [1c85032d] Removing vds [] from the domain in maintenance cache
>2014-08-03 04:51:52,823 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [1c85032d] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 04:51:52,825 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-14) [8557062] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 04:51:55,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 18e5c347
>2014-08-03 04:51:56,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] FINISH, GetHardwareInfoVDSCommand, log id: 18e5c347
>2014-08-03 04:51:56,053 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [5b8094aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 04:51:56,068 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [11555cf2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 04:51:56,078 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 04:51:56,090 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 314f5dea
>2014-08-03 04:51:56,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 314f5dea
>2014-08-03 04:51:56,151 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] Host rack storage connection was succeeded 
>2014-08-03 04:51:56,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 46b0eb76
>2014-08-03 04:52:02,524 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 04:52:02,528 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 04:52:56,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 46b0eb76
>2014-08-03 04:52:56,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 66590958
>2014-08-03 04:52:56,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 66590958
>2014-08-03 04:52:56,709 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [18923d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 04:52:56,717 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [18923d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 04:52:56,740 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 04:53:02,727 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 04:53:02,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 04:53:02,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40d4557f
>2014-08-03 04:53:02,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling started: taskId = 30cf0799-5a02-40eb-99c2-41c284129530
>2014-08-03 04:53:03,794 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended: taskId = 30cf0799-5a02-40eb-99c2-41c284129530 task status = finished
>2014-08-03 04:53:03,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended, spm status: SPM
>2014-08-03 04:53:03,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=30cf0799-5a02-40eb-99c2-41c284129530), log id: 1e496c0f
>2014-08-03 04:53:03,896 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, HSMClearTaskVDSCommand, log id: 1e496c0f
>2014-08-03 04:53:03,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@439b8a20, log id: 40d4557f
>2014-08-03 04:53:03,901 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 04:53:03,907 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 04:53:03,917 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 18dc5e46
>2014-08-03 04:53:03,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 04:53:03,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7066ebad
>2014-08-03 04:53:03,960 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7066ebad
>2014-08-03 04:53:03,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 18dc5e46
>2014-08-03 04:53:03,962 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool Default
>2014-08-03 05:23:55,778 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized
>2014-08-03 05:24:23,877 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [161c5527] Attempting to update VMs/Templates Ovf.
>2014-08-03 06:23:55,785 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized
>2014-08-03 06:24:23,360 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-36) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 06:24:23,894 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf.
>2014-08-03 07:23:55,791 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [71be4699] Principal admin::internal synchronized
>2014-08-03 07:24:23,908 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf.
>2014-08-03 07:52:51,484 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 07:52:51,486 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 07:54:18,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 07:54:18,527 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 07:54:18,531 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 07:54:18,533 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 07:54:18,549 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Correlation ID: 61a1f9b5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 07:56:18,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 07:56:18,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 07:56:18,580 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 07:56:18,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 07:56:18,621 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 07:56:18,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: d41f226
>2014-08-03 07:56:18,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1b1e2478
>2014-08-03 07:56:18,652 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 07:58:18,546 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 07:58:18,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] FINISH, SpmStopVDSCommand, log id: 1b1e2478
>2014-08-03 07:58:18,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 07:58:18,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 07:58:18,560 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 07:58:18,563 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] FINISH, ResetIrsVDSCommand, log id: d41f226
>2014-08-03 07:58:18,561 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 07:58:18,563 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 07:58:18,577 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [242d9d56] Correlation ID: 242d9d56, Job ID: c1a40082-c940-4518-b450-ec68c25d1d62, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 07:58:18,603 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 07:58:18,607 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [61a1f9b5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 07:58:18,609 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 07:58:18,680 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 07:58:18,703 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 07:58:18,710 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 07:58:18,717 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 07:58:18,728 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [536db155] Correlation ID: 536db155, Job ID: a595a0b5-bf76-45ee-b43c-436637c60038, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 07:58:18,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [536db155] Removing vds [] from the domain in maintenance cache
>2014-08-03 07:58:18,735 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [536db155] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 07:58:18,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 07:58:18,741 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 07:58:18,766 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 07:58:18,774 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] Removing vds [] from the domain in maintenance cache
>2014-08-03 07:58:18,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 07:58:18,776 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [61a1f9b5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 07:58:21,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4d09556a
>2014-08-03 07:58:21,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GetHardwareInfoVDSCommand, log id: 4d09556a
>2014-08-03 07:58:21,963 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-1) [6b7975ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 07:58:21,973 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-1) [6dd1a1f8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 07:58:21,980 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 07:58:21,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 755a37c7
>2014-08-03 07:58:22,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 755a37c7
>2014-08-03 07:58:22,057 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Host rack storage connection was succeeded 
>2014-08-03 07:58:22,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6b85978e
>2014-08-03 07:58:28,570 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 07:58:28,574 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 07:59:22,463 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 6b85978e
>2014-08-03 07:59:22,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 73c51cbb
>2014-08-03 07:59:22,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] FINISH, SetMOMPolicyParametersVDSCommand, log id: 73c51cbb
>2014-08-03 07:59:22,578 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 07:59:22,584 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 07:59:22,599 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-1) [795e8140] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 07:59:28,725 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 07:59:28,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 07:59:28,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47334e56
>2014-08-03 07:59:28,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling started: taskId = 3c17a062-6669-495e-8dbb-5b48f9b758d4
>2014-08-03 07:59:29,789 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling ended: taskId = 3c17a062-6669-495e-8dbb-5b48f9b758d4 task status = finished
>2014-08-03 07:59:29,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling ended, spm status: SPM
>2014-08-03 07:59:29,848 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=3c17a062-6669-495e-8dbb-5b48f9b758d4), log id: 1e65786d
>2014-08-03 07:59:29,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] FINISH, HSMClearTaskVDSCommand, log id: 1e65786d
>2014-08-03 07:59:29,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@234cd6d3, log id: 47334e56
>2014-08-03 07:59:29,898 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 07:59:29,905 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 07:59:29,915 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f7423dd
>2014-08-03 07:59:29,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 07:59:29,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7e4e0a7c
>2014-08-03 07:59:29,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e4e0a7c
>2014-08-03 07:59:29,963 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f7423dd
>2014-08-03 07:59:29,964 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] Discovered no tasks on Storage Pool Default
>2014-08-03 08:23:55,797 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [78845858] Principal admin::internal synchronized
>2014-08-03 08:24:23,921 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [81f884c] Attempting to update VMs/Templates Ovf.
>2014-08-03 09:23:55,806 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized
>2014-08-03 09:24:23,954 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf.
>2014-08-03 10:23:55,812 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized
>2014-08-03 10:24:23,969 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-08-03 10:59:11,542 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 10:59:11,544 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 11:00:45,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [593e74a7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 11:00:45,462 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 11:00:45,466 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 11:00:45,468 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 11:00:45,479 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Correlation ID: 5f00c7bf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 11:02:45,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 11:02:45,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 11:02:45,515 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 11:02:45,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 11:02:45,551 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 11:02:45,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 612569f0
>2014-08-03 11:02:45,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2458fd84
>2014-08-03 11:02:45,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 11:04:45,480 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 11:04:45,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] FINISH, SpmStopVDSCommand, log id: 2458fd84
>2014-08-03 11:04:45,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-23) [1802a60f] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 11:04:45,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 11:04:45,497 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [1802a60f] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 11:04:45,499 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] FINISH, ResetIrsVDSCommand, log id: 612569f0
>2014-08-03 11:04:45,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 11:04:45,500 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [1802a60f] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 11:04:45,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [1802a60f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 11:04:45,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [287a2335] Correlation ID: 287a2335, Job ID: e32f59b0-6a5e-4c1f-9e0d-32ec64fb4f45, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 11:04:45,542 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [5f00c7bf] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 11:04:45,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 11:04:45,551 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [1802a60f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 11:04:45,613 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 11:04:45,632 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 11:04:45,639 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] Removing vds [] from the domain in maintenance cache
>2014-08-03 11:04:45,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 11:04:45,641 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [5f00c7bf] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 11:04:45,643 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 11:04:45,648 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 11:04:45,671 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 11:04:45,679 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 11:04:45,687 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 11:04:45,700 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Correlation ID: 6f57b46c, Job ID: c59687b5-9359-421d-92bd-1cc5b1fb4a57, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 11:04:45,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Removing vds [] from the domain in maintenance cache
>2014-08-03 11:04:45,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 11:04:48,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 584b8f03
>2014-08-03 11:04:48,901 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GetHardwareInfoVDSCommand, log id: 584b8f03
>2014-08-03 11:04:48,926 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [7c2b8d14] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 11:04:48,976 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [2bc952cd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 11:04:48,984 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 11:04:48,997 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ab136d4
>2014-08-03 11:04:49,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7ab136d4
>2014-08-03 11:04:49,058 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] Host rack storage connection was succeeded 
>2014-08-03 11:04:49,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3eaf384d
>2014-08-03 11:04:55,507 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 11:04:55,511 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 11:05:49,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, ConnectStoragePoolVDSCommand, log id: 3eaf384d
>2014-08-03 11:05:49,576 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 77df883c
>2014-08-03 11:05:49,669 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 77df883c
>2014-08-03 11:05:49,677 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [34c7414c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 11:05:49,683 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [34c7414c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 11:05:49,699 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [56fe64d8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 11:05:55,656 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 11:05:55,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 11:05:55,665 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78f11151
>2014-08-03 11:05:55,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = b00c45e9-c194-40dd-aca3-461bc8ed5ab7
>2014-08-03 11:05:56,720 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = b00c45e9-c194-40dd-aca3-461bc8ed5ab7 task status = finished
>2014-08-03 11:05:56,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM
>2014-08-03 11:05:56,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b00c45e9-c194-40dd-aca3-461bc8ed5ab7), log id: 73344168
>2014-08-03 11:05:56,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 73344168
>2014-08-03 11:05:56,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d023b8a, log id: 78f11151
>2014-08-03 11:05:56,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 11:05:56,836 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 11:05:56,847 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6594af2f
>2014-08-03 11:05:56,876 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 11:05:56,878 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 659a1882
>2014-08-03 11:05:56,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 659a1882
>2014-08-03 11:05:56,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6594af2f
>2014-08-03 11:05:56,886 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default
>2014-08-03 11:23:55,817 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-03 11:24:24,005 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf.
>2014-08-03 12:23:55,822 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [519c108f] Principal admin::internal synchronized
>2014-08-03 12:24:24,025 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf.
>2014-08-03 12:53:24,194 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 13:23:55,830 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized
>2014-08-03 13:24:24,052 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Attempting to update VMs/Templates Ovf.
>2014-08-03 14:05:44,214 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 14:05:44,216 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 14:07:13,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 14:07:13,587 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 14:07:13,590 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 14:07:13,592 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 14:07:13,604 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Correlation ID: 708ffdc9, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 14:09:13,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 14:09:13,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) [708ffdc9] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 14:09:13,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 14:09:13,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 14:09:13,685 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 14:09:13,705 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3c1fa393
>2014-08-03 14:09:13,709 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 695e1ab0
>2014-08-03 14:09:13,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 14:10:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Autorecovering 1 storage domains
>2014-08-03 14:10:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-03 14:10:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 14:10:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 2:10 PM
>2014-08-03 14:10:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-03 14:10:00,024 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74dc7bbc
>2014-08-03 14:11:13,605 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 14:11:13,610 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 14:11:13,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] FINISH, SpmStopVDSCommand, log id: 695e1ab0
>2014-08-03 14:11:13,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 14:11:13,613 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [5aa74089] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 14:11:13,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] FINISH, ConnectStorageServerVDSCommand, log id: 74dc7bbc
>2014-08-03 14:11:13,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 14:11:13,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 14:11:13,623 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-03 14:11:13,622 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] FINISH, ResetIrsVDSCommand, log id: 3c1fa393
>2014-08-03 14:11:13,619 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [5aa74089] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 14:11:13,636 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [3b915adf] Correlation ID: 3b915adf, Job ID: 668cbbdf-0b1a-4927-b758-2c8b93beded3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 14:11:13,631 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-03 14:11:13,658 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 2:11 PM
>2014-08-03 14:11:13,659 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [5aa74089] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 14:11:13,658 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [708ffdc9] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 14:11:13,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 14:11:13,724 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 14:11:13,743 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 14:11:13,750 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 14:11:13,767 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 14:11:13,777 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Correlation ID: 17ef6b50, Job ID: f8272be9-841f-47fe-9989-00ed56f47cbd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 14:11:13,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Removing vds [] from the domain in maintenance cache
>2014-08-03 14:11:13,783 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 14:11:13,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 14:11:13,790 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 14:11:13,807 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-37) [66828990] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 14:11:13,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [66828990] Removing vds [] from the domain in maintenance cache
>2014-08-03 14:11:13,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [66828990] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 14:11:13,816 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [708ffdc9] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 14:11:17,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 7fc96f76
>2014-08-03 14:11:17,221 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GetHardwareInfoVDSCommand, log id: 7fc96f76
>2014-08-03 14:11:17,245 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [2e7ea298] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 14:11:17,295 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [5088e5e4] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 14:11:17,303 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 14:11:17,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 78297031
>2014-08-03 14:11:17,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 78297031
>2014-08-03 14:11:17,380 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Host rack storage connection was succeeded 
>2014-08-03 14:11:17,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 247bfb93
>2014-08-03 14:11:23,628 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 14:11:23,632 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 14:12:17,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 247bfb93
>2014-08-03 14:12:17,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 910d1b3
>2014-08-03 14:12:17,919 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] FINISH, SetMOMPolicyParametersVDSCommand, log id: 910d1b3
>2014-08-03 14:12:17,927 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 14:12:17,936 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 14:12:17,951 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [275ae126] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 14:12:23,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 14:12:23,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 14:12:23,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2730a707
>2014-08-03 14:12:23,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling started: taskId = 515c9f8e-e8c2-4f45-bbc1-b9fc292222ce
>2014-08-03 14:12:24,809 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling ended: taskId = 515c9f8e-e8c2-4f45-bbc1-b9fc292222ce task status = finished
>2014-08-03 14:12:24,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling ended, spm status: SPM
>2014-08-03 14:12:24,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=515c9f8e-e8c2-4f45-bbc1-b9fc292222ce), log id: 7ce97c7b
>2014-08-03 14:12:24,912 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] FINISH, HSMClearTaskVDSCommand, log id: 7ce97c7b
>2014-08-03 14:12:24,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a65aa9f, log id: 2730a707
>2014-08-03 14:12:24,918 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 14:12:24,925 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [16080eb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 14:12:24,926 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 14:12:24,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3a1df1b3
>2014-08-03 14:12:24,965 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 14:12:24,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 48ebd35a
>2014-08-03 14:12:24,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48ebd35a
>2014-08-03 14:12:24,974 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a1df1b3
>2014-08-03 14:12:24,975 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [16080eb6] Discovered no tasks on Storage Pool Default
>2014-08-03 14:23:55,837 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) [6452f878] Principal admin::internal synchronized
>2014-08-03 14:24:24,068 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-08-03 15:23:55,843 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-08-03 15:24:24,083 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf.
>2014-08-03 16:23:55,849 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) [42c2440e] Principal admin::internal synchronized
>2014-08-03 16:24:24,097 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-08-03 17:12:03,451 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 17:12:18,572 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 17:13:43,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 17:13:43,124 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 17:13:43,128 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [1abadf13] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 17:13:43,131 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [1abadf13] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 17:13:43,147 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [1abadf13] Correlation ID: 1abadf13, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 17:15:43,145 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 17:15:43,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [1abadf13] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 17:15:43,148 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 17:15:43,149 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 17:15:43,185 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 17:15:43,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3e0dce79
>2014-08-03 17:15:43,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 55f492d4
>2014-08-03 17:15:43,218 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 17:17:43,154 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 17:17:43,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 17:17:43,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 17:17:43,162 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [5e890dae] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 17:17:43,160 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] FINISH, SpmStopVDSCommand, log id: 55f492d4
>2014-08-03 17:17:43,166 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 17:17:43,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [5e890dae] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 17:17:43,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] FINISH, ResetIrsVDSCommand, log id: 3e0dce79
>2014-08-03 17:17:43,200 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [5e890dae] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 17:17:43,209 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [513e5878] Correlation ID: 513e5878, Job ID: bb490939-775a-45b0-a39f-6a991a569306, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 17:17:43,217 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [1abadf13] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 17:17:43,219 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 17:17:43,279 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 17:17:43,297 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 17:17:43,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] Removing vds [] from the domain in maintenance cache
>2014-08-03 17:17:43,307 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 17:17:43,308 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [1abadf13] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 17:17:43,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 17:17:43,315 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 17:17:43,329 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 17:17:43,336 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 17:17:43,341 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 17:17:43,351 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [319458de] Correlation ID: 319458de, Job ID: 9e46c5f7-417b-4e95-981c-62ecab05322b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 17:17:43,356 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [319458de] Removing vds [] from the domain in maintenance cache
>2014-08-03 17:17:43,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [319458de] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 17:17:46,460 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 692d70c1
>2014-08-03 17:17:46,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, GetHardwareInfoVDSCommand, log id: 692d70c1
>2014-08-03 17:17:46,550 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [76a195c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 17:17:46,599 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [1603d8f1] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 17:17:46,607 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 17:17:46,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3d4a582
>2014-08-03 17:17:46,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3d4a582
>2014-08-03 17:17:46,679 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Host rack storage connection was succeeded 
>2014-08-03 17:17:46,687 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5b56f9b7
>2014-08-03 17:17:53,176 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 17:17:53,181 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 17:18:47,078 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 5b56f9b7
>2014-08-03 17:18:47,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23076021
>2014-08-03 17:18:47,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 23076021
>2014-08-03 17:18:47,221 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 17:18:47,227 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 17:18:47,268 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 17:18:53,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 17:18:53,371 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 17:18:53,374 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15876e44
>2014-08-03 17:18:53,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling started: taskId = e0001b88-a74f-4c57-92c8-3f51c0067d16
>2014-08-03 17:18:54,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling ended: taskId = e0001b88-a74f-4c57-92c8-3f51c0067d16 task status = finished
>2014-08-03 17:18:54,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling ended, spm status: SPM
>2014-08-03 17:18:54,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=e0001b88-a74f-4c57-92c8-3f51c0067d16), log id: 3002d464
>2014-08-03 17:18:54,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] FINISH, HSMClearTaskVDSCommand, log id: 3002d464
>2014-08-03 17:18:54,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b24b935, log id: 15876e44
>2014-08-03 17:18:54,543 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 17:18:54,552 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 17:18:54,555 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 17:18:54,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 23e4f08b
>2014-08-03 17:18:54,596 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 17:18:54,598 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5736c62f
>2014-08-03 17:18:54,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5736c62f
>2014-08-03 17:18:54,605 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23e4f08b
>2014-08-03 17:18:54,605 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [782c75f] Discovered no tasks on Storage Pool Default
>2014-08-03 17:23:55,855 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized
>2014-08-03 17:24:24,130 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Attempting to update VMs/Templates Ovf.
>2014-08-03 18:23:55,862 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) [39225a1e] Principal admin::internal synchronized
>2014-08-03 18:24:24,145 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf.
>2014-08-03 19:23:55,869 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-48) [14cac44e] Principal admin::internal synchronized
>2014-08-03 19:24:24,160 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Attempting to update VMs/Templates Ovf.
>2014-08-03 20:18:39,494 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 20:18:39,496 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 20:20:14,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 20:20:14,601 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 20:20:14,605 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 20:20:14,607 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 20:20:14,619 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Correlation ID: 3e29dd44, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 20:22:14,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 20:22:14,653 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [3e29dd44] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 20:22:14,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 20:22:14,655 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 20:22:14,691 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 20:22:14,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3b956c06
>2014-08-03 20:22:14,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1f25fce3
>2014-08-03 20:22:14,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 20:23:55,875 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-08-03 20:24:14,621 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 20:24:14,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] FINISH, SpmStopVDSCommand, log id: 1f25fce3
>2014-08-03 20:24:14,623 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 20:24:14,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 20:24:14,630 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 20:24:14,631 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 20:24:14,633 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] FINISH, ResetIrsVDSCommand, log id: 3b956c06
>2014-08-03 20:24:14,633 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 20:24:14,636 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 20:24:14,643 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [1b05141c] Correlation ID: 1b05141c, Job ID: 3489d1c6-b451-41d5-a2e1-c407c3516776, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 20:24:14,664 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [3e29dd44] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 20:24:14,665 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 20:24:14,666 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 20:24:14,738 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 20:24:14,756 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [80dee22] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 20:24:14,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [80dee22] Removing vds [] from the domain in maintenance cache
>2014-08-03 20:24:14,765 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [80dee22] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 20:24:14,767 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [3e29dd44] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 20:24:14,770 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 20:24:14,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 20:24:14,792 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 20:24:14,798 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 20:24:14,804 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 20:24:14,813 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Correlation ID: 6054abd1, Job ID: 1358fee6-c04d-4e99-9b94-0ebe8c5d5189, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 20:24:14,818 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Removing vds [] from the domain in maintenance cache
>2014-08-03 20:24:14,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 20:24:17,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 157ad5c1
>2014-08-03 20:24:17,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] FINISH, GetHardwareInfoVDSCommand, log id: 157ad5c1
>2014-08-03 20:24:18,005 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [78587bc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 20:24:18,057 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [139f8621] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 20:24:18,064 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 20:24:18,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5845eeec
>2014-08-03 20:24:18,130 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5845eeec
>2014-08-03 20:24:18,131 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] Host rack storage connection was succeeded 
>2014-08-03 20:24:18,137 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 477818a3
>2014-08-03 20:24:24,182 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Attempting to update VMs/Templates Ovf.
>2014-08-03 20:24:24,638 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-1) [795e8140] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 20:24:24,644 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-1) [795e8140] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 20:25:18,565 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 477818a3
>2014-08-03 20:25:18,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4790eeda
>2014-08-03 20:25:18,697 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4790eeda
>2014-08-03 20:25:18,705 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [66f70bde] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 20:25:18,711 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [66f70bde] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 20:25:18,730 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [3adcf236] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 20:25:24,808 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 20:25:24,814 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 20:25:24,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 177fac0c
>2014-08-03 20:25:24,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling started: taskId = 6a664af8-94a4-46f6-8bb6-8285a696ad10
>2014-08-03 20:25:25,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling ended: taskId = 6a664af8-94a4-46f6-8bb6-8285a696ad10 task status = finished
>2014-08-03 20:25:25,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling ended, spm status: SPM
>2014-08-03 20:25:25,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6a664af8-94a4-46f6-8bb6-8285a696ad10), log id: 33bb9374
>2014-08-03 20:25:25,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] FINISH, HSMClearTaskVDSCommand, log id: 33bb9374
>2014-08-03 20:25:25,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@686dcc01, log id: 177fac0c
>2014-08-03 20:25:25,980 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 20:25:25,988 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 20:25:25,989 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 20:25:25,998 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 20b30b90
>2014-08-03 20:25:26,040 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 20:25:26,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 351b6555
>2014-08-03 20:25:26,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 351b6555
>2014-08-03 20:25:26,048 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20b30b90
>2014-08-03 20:25:26,049 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] Discovered no tasks on Storage Pool Default
>2014-08-03 20:27:16,796 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 20:27:16,797 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 21:23:55,881 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-45) [22578e65] Principal admin::internal synchronized
>2014-08-03 21:24:24,185 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf.
>2014-08-03 22:23:55,886 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-08-03 22:24:24,203 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [42c2440e] Attempting to update VMs/Templates Ovf.
>2014-08-03 23:23:55,893 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Principal admin::internal synchronized
>2014-08-03 23:24:24,218 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [81f884c] Attempting to update VMs/Templates Ovf.
>2014-08-03 23:25:13,035 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-03 23:25:13,037 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-03 23:26:49,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 23:26:49,209 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 23:26:49,214 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-03 23:26:49,219 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-03 23:26:49,234 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Correlation ID: 6bf204c4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-03 23:28:49,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 23:28:49,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) [6bf204c4] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 23:28:49,252 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-03 23:28:49,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-03 23:28:49,305 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 23:28:49,331 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 310d312f
>2014-08-03 23:28:49,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2d32a150
>2014-08-03 23:28:49,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-03 23:30:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2e3238db] Autorecovering 1 storage domains
>2014-08-03 23:30:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2e3238db] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-03 23:30:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-03 23:30:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 11:30 PM
>2014-08-03 23:30:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-03 23:30:00,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 33b2f788
>2014-08-03 23:30:49,218 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-03 23:30:49,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-03 23:30:49,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 23:30:49,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] FINISH, ConnectStorageServerVDSCommand, log id: 33b2f788
>2014-08-03 23:30:49,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-56) [65d9de58] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 23:30:49,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] FINISH, SpmStopVDSCommand, log id: 2d32a150
>2014-08-03 23:30:49,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-03 23:30:49,236 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [65d9de58] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-03 23:30:49,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [65d9de58] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-03 23:30:49,233 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-03 23:30:49,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [65d9de58] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-03 23:30:49,245 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-03 23:30:49,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] FINISH, ResetIrsVDSCommand, log id: 310d312f
>2014-08-03 23:30:49,285 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 11:30 PM
>2014-08-03 23:30:49,288 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [65d9de58] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-03 23:30:49,295 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [58abd855] Correlation ID: 58abd855, Job ID: 2a69ce69-cc51-4748-a61d-3aaa4e35b40b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-03 23:30:49,303 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6bf204c4] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 23:30:49,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-03 23:30:49,371 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-03 23:30:49,386 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 23:30:49,393 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-03 23:30:49,398 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-03 23:30:49,409 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Correlation ID: 1879c3dc, Job ID: 60a48859-b76f-4664-a92c-078104fbf974, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-03 23:30:49,416 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Removing vds [] from the domain in maintenance cache
>2014-08-03 23:30:49,417 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 23:30:49,419 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-03 23:30:49,424 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-03 23:30:49,441 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-03 23:30:49,448 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] Removing vds [] from the domain in maintenance cache
>2014-08-03 23:30:49,449 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] Removing host(s) [] from hosts unseen domain report cache
>2014-08-03 23:30:49,449 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6bf204c4] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-03 23:30:52,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6828084c
>2014-08-03 23:30:52,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] HostName = rack
>2014-08-03 23:30:52,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-03 23:30:52,377 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-03 23:30:52,386 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-03 23:30:55,619 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ffaf88c
>2014-08-03 23:30:55,671 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] FINISH, GetHardwareInfoVDSCommand, log id: ffaf88c
>2014-08-03 23:30:55,698 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [728a99d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 23:30:55,747 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) [728a99d4] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-03 23:30:55,749 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [3fb3de80] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 23:30:55,758 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-03 23:30:55,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 147481ae
>2014-08-03 23:30:55,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 147481ae
>2014-08-03 23:30:55,837 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] Host rack storage connection was succeeded 
>2014-08-03 23:30:55,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 21582090
>2014-08-03 23:30:59,244 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 23:30:59,248 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-03 23:31:56,816 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 21582090
>2014-08-03 23:31:56,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: b1fe2a2
>2014-08-03 23:31:56,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] FINISH, SetMOMPolicyParametersVDSCommand, log id: b1fe2a2
>2014-08-03 23:31:56,934 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [571f42d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-03 23:31:56,941 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [571f42d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-03 23:31:56,958 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-03 23:31:59,369 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-03 23:31:59,374 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-03 23:31:59,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40cfa0a4
>2014-08-03 23:31:59,425 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = da4e92a9-37ec-46cf-bf9d-731665d0aec0
>2014-08-03 23:32:00,431 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = da4e92a9-37ec-46cf-bf9d-731665d0aec0 task status = finished
>2014-08-03 23:32:00,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM
>2014-08-03 23:32:00,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=da4e92a9-37ec-46cf-bf9d-731665d0aec0), log id: 7b314fd5
>2014-08-03 23:32:00,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 7b314fd5
>2014-08-03 23:32:00,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47478612, log id: 40cfa0a4
>2014-08-03 23:32:00,550 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-03 23:32:00,557 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-03 23:32:00,559 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-03 23:32:00,571 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4e41b38c
>2014-08-03 23:32:00,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-03 23:32:00,617 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 66a4eee0
>2014-08-03 23:32:00,623 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66a4eee0
>2014-08-03 23:32:00,624 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e41b38c
>2014-08-03 23:32:00,625 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) [4760e42] Discovered no tasks on Storage Pool Default
>2014-08-04 00:23:55,906 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized
>2014-08-04 00:24:24,234 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Attempting to update VMs/Templates Ovf.
>2014-08-04 01:23:55,912 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-08-04 01:24:24,251 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf.
>2014-08-04 02:23:55,919 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized
>2014-08-04 02:24:24,263 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf.
>2014-08-04 02:31:49,088 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 02:32:04,224 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 02:33:27,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 02:33:27,941 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 02:33:27,945 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 02:33:27,947 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 02:33:27,960 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Correlation ID: 4f7e782e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 02:35:27,990 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 02:35:27,992 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [4f7e782e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 02:35:27,993 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 02:35:27,994 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 02:35:28,045 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 02:35:28,066 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5dc8bb9b
>2014-08-04 02:35:28,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3954a733
>2014-08-04 02:35:28,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 02:37:27,961 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 02:37:27,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 02:37:27,970 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 02:37:27,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 02:37:27,973 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7526d482] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 02:37:27,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] FINISH, SpmStopVDSCommand, log id: 3954a733
>2014-08-04 02:37:27,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7526d482] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 02:37:27,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] FINISH, ResetIrsVDSCommand, log id: 5dc8bb9b
>2014-08-04 02:37:28,014 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7526d482] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 02:37:28,019 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] Correlation ID: c9f7d42, Job ID: 8de3de07-7c75-47d5-8365-56a770348bad, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 02:37:28,025 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [4f7e782e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 02:37:28,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 02:37:28,085 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 02:37:28,103 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64520b65] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 02:37:28,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64520b65] Removing vds [] from the domain in maintenance cache
>2014-08-04 02:37:28,110 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64520b65] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 02:37:28,111 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [4f7e782e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 02:37:28,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 02:37:28,118 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 02:37:28,133 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 02:37:28,139 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 02:37:28,145 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 02:37:28,157 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Correlation ID: 3eb0d362, Job ID: 5e8b9990-6fd8-4a56-98f0-b0349c058b39, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 02:37:28,163 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Removing vds [] from the domain in maintenance cache
>2014-08-04 02:37:28,164 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 02:37:31,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2809d9c7
>2014-08-04 02:37:31,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] FINISH, GetHardwareInfoVDSCommand, log id: 2809d9c7
>2014-08-04 02:37:31,345 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [5165d85a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 02:37:31,394 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [2783c8a8] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 02:37:31,400 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 02:37:31,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1f49a2d3
>2014-08-04 02:37:31,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1f49a2d3
>2014-08-04 02:37:31,470 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] Host rack storage connection was succeeded 
>2014-08-04 02:37:31,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bf13231
>2014-08-04 02:37:37,974 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 02:37:37,985 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 02:38:31,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, ConnectStoragePoolVDSCommand, log id: 2bf13231
>2014-08-04 02:38:31,903 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7497473
>2014-08-04 02:38:32,049 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7497473
>2014-08-04 02:38:32,056 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [760b6984] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 02:38:32,062 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [760b6984] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 02:38:32,078 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 02:38:38,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 02:38:38,145 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 02:38:38,148 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5fa85992
>2014-08-04 02:38:38,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling started: taskId = cad5e411-9896-4abf-850d-6caea819d97e
>2014-08-04 02:38:39,203 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended: taskId = cad5e411-9896-4abf-850d-6caea819d97e task status = finished
>2014-08-04 02:38:39,275 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended, spm status: SPM
>2014-08-04 02:38:39,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cad5e411-9896-4abf-850d-6caea819d97e), log id: 73e43832
>2014-08-04 02:38:39,325 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, HSMClearTaskVDSCommand, log id: 73e43832
>2014-08-04 02:38:39,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a94e565, log id: 5fa85992
>2014-08-04 02:38:39,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 02:38:39,393 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1b83a377] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 02:38:39,394 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 02:38:39,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 69415df4
>2014-08-04 02:38:39,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 02:38:39,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 518890f4
>2014-08-04 02:38:39,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 518890f4
>2014-08-04 02:38:39,477 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69415df4
>2014-08-04 02:38:39,477 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [1b83a377] Discovered no tasks on Storage Pool Default
>2014-08-04 03:23:55,927 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized
>2014-08-04 03:24:24,298 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) [686e64a1] Attempting to update VMs/Templates Ovf.
>2014-08-04 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Start deleteAgedOutAuditLogs
>2014-08-04 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Start deleteAgedOutCommandEntities
>2014-08-04 03:35:35,006 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Finished deleteAgedOutAuditLogs
>2014-08-04 03:35:35,008 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Finished deleteAgedOutCommandEntities
>2014-08-04 04:23:55,934 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized
>2014-08-04 04:24:24,332 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf.
>2014-08-04 05:23:55,940 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Principal admin::internal synchronized
>2014-08-04 05:24:24,345 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-08-04 05:38:38,371 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 05:38:38,373 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 05:40:08,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 05:40:08,122 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 05:40:08,124 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [32fc3079] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 05:40:08,126 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [32fc3079] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 05:40:08,136 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [32fc3079] Correlation ID: 32fc3079, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 05:42:08,169 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 05:42:08,173 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [32fc3079] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 05:42:08,174 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 05:42:08,175 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 05:42:08,202 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 05:42:08,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3cc190e4
>2014-08-04 05:42:08,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3f9e1a8c
>2014-08-04 05:42:08,234 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 05:44:08,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 05:44:08,144 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-2) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 05:44:08,142 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 05:44:08,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] FINISH, SpmStopVDSCommand, log id: 3f9e1a8c
>2014-08-04 05:44:08,151 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 05:44:08,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [55405a19] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 05:44:08,152 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 05:44:08,160 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] FINISH, ResetIrsVDSCommand, log id: 3cc190e4
>2014-08-04 05:44:08,190 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 05:44:08,195 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] Correlation ID: 2ae4fdd9, Job ID: 55b2cd1f-f392-44ab-95c4-85d932d58c98, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 05:44:08,203 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-8) [32fc3079] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 05:44:08,205 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 05:44:08,280 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 05:44:08,305 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-31) [37841764] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 05:44:08,312 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) [37841764] Removing vds [] from the domain in maintenance cache
>2014-08-04 05:44:08,313 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) [37841764] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 05:44:08,314 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-8) [32fc3079] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 05:44:08,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 05:44:08,322 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 05:44:08,338 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 05:44:08,345 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 05:44:08,351 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 05:44:08,361 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Correlation ID: 283c9aab, Job ID: c912c298-b426-4567-82d9-401dbefb5195, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 05:44:08,366 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Removing vds [] from the domain in maintenance cache
>2014-08-04 05:44:08,367 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 05:44:11,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [778da158] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 73e0104a
>2014-08-04 05:44:11,504 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [778da158] FINISH, GetHardwareInfoVDSCommand, log id: 73e0104a
>2014-08-04 05:44:11,531 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [4e3a2091] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 05:44:11,581 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [5561bbe6] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 05:44:11,589 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 05:44:11,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e5be5aa
>2014-08-04 05:44:11,663 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e5be5aa
>2014-08-04 05:44:11,664 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Host rack storage connection was succeeded 
>2014-08-04 05:44:11,675 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: a77f362
>2014-08-04 05:44:18,156 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 05:44:18,161 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 05:45:12,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: a77f362
>2014-08-04 05:45:12,328 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1a63527c
>2014-08-04 05:45:12,423 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1a63527c
>2014-08-04 05:45:12,430 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 05:45:12,436 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 05:45:12,452 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [2cd68be0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 05:45:18,280 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 05:45:18,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 05:45:18,288 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dd3b127
>2014-08-04 05:45:18,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling started: taskId = 726e6e7a-639c-4042-bbea-8c35312f43ee
>2014-08-04 05:45:19,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling ended: taskId = 726e6e7a-639c-4042-bbea-8c35312f43ee task status = finished
>2014-08-04 05:45:19,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling ended, spm status: SPM
>2014-08-04 05:45:19,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=726e6e7a-639c-4042-bbea-8c35312f43ee), log id: 30ffa7ff
>2014-08-04 05:45:19,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, HSMClearTaskVDSCommand, log id: 30ffa7ff
>2014-08-04 05:45:19,450 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2995aa21, log id: 2dd3b127
>2014-08-04 05:45:19,455 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 05:45:19,463 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [81f884c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 05:45:19,465 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-18) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 05:45:19,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32d8dd7b
>2014-08-04 05:45:19,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 05:45:19,505 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3bf0d2ea
>2014-08-04 05:45:19,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bf0d2ea
>2014-08-04 05:45:19,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32d8dd7b
>2014-08-04 05:45:19,514 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-18) [81f884c] Discovered no tasks on Storage Pool Default
>2014-08-04 06:23:55,946 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized
>2014-08-04 06:24:24,360 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf.
>2014-08-04 07:23:55,952 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized
>2014-08-04 07:24:24,376 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Attempting to update VMs/Templates Ovf.
>2014-08-04 08:14:49,659 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 08:23:55,962 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [519c108f] Principal admin::internal synchronized
>2014-08-04 08:24:24,410 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [4341002] Attempting to update VMs/Templates Ovf.
>2014-08-04 08:45:13,493 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 08:45:13,495 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 08:46:40,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 08:46:40,494 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 08:46:40,498 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 08:46:40,500 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 08:46:40,515 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Correlation ID: 6a286cf6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 08:48:40,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 08:48:40,544 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [6a286cf6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 08:48:40,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 08:48:40,547 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 08:48:40,582 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 08:48:40,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 71044258
>2014-08-04 08:48:40,604 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 49dccb43
>2014-08-04 08:48:40,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 08:50:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering 1 storage domains
>2014-08-04 08:50:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-04 08:50:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 08:50:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/4/14 8:50 AM
>2014-08-04 08:50:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-04 08:50:00,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 787b77bb
>2014-08-04 08:50:40,512 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 08:50:40,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] FINISH, SpmStopVDSCommand, log id: 49dccb43
>2014-08-04 08:50:40,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [4341002] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 08:50:40,520 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [4341002] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 08:50:40,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 08:50:40,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 08:50:40,526 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] FINISH, ConnectStorageServerVDSCommand, log id: 787b77bb
>2014-08-04 08:50:40,527 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-04 08:50:40,523 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 08:50:40,521 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [4341002] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 08:50:40,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] FINISH, ResetIrsVDSCommand, log id: 71044258
>2014-08-04 08:50:40,532 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-04 08:50:40,555 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/4/14 8:50 AM
>2014-08-04 08:50:40,562 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [4341002] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 08:50:40,567 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] Correlation ID: 69d6e1dd, Job ID: 9969f4b5-f80c-46ef-8f13-1be65ba93ce9, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 08:50:40,575 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [6a286cf6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 08:50:40,577 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 08:50:40,652 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 08:50:40,679 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [821b4e7] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 08:50:40,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [821b4e7] Removing vds [] from the domain in maintenance cache
>2014-08-04 08:50:40,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [821b4e7] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 08:50:40,690 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [6a286cf6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 08:50:40,692 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 08:50:40,698 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 08:50:40,713 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 08:50:40,720 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 08:50:40,727 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 08:50:40,737 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Correlation ID: 60237d1f, Job ID: bd045107-439b-4448-9f13-33fac5371203, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 08:50:40,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Removing vds [] from the domain in maintenance cache
>2014-08-04 08:50:40,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 08:50:43,657 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@793c69e1
>2014-08-04 08:50:43,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] HostName = rack
>2014-08-04 08:50:43,660 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-04 08:50:43,674 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [32fc3079] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-04 08:50:43,680 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [32fc3079] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-04 08:50:46,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 45e49805
>2014-08-04 08:50:46,961 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] FINISH, GetHardwareInfoVDSCommand, log id: 45e49805
>2014-08-04 08:50:46,984 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [156f0be3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 08:50:47,033 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [156f0be3] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-04 08:50:47,035 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-85) [59ba3d1e] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 08:50:47,043 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 08:50:47,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1cbfa1dc
>2014-08-04 08:50:47,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1cbfa1dc
>2014-08-04 08:50:47,120 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] Host rack storage connection was succeeded 
>2014-08-04 08:50:47,129 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 65528e10
>2014-08-04 08:50:50,534 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 08:50:50,537 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 08:51:47,520 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 65528e10
>2014-08-04 08:51:47,538 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 79ef92b2
>2014-08-04 08:51:47,634 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] FINISH, SetMOMPolicyParametersVDSCommand, log id: 79ef92b2
>2014-08-04 08:51:47,645 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [149b0156] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 08:51:47,653 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [149b0156] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 08:51:47,671 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [9016fac] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 08:51:50,668 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 08:51:50,675 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 08:51:50,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66cfa6c3
>2014-08-04 08:51:50,729 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling started: taskId = 456c1d7c-a0e4-4842-8e00-90faa50e28ef
>2014-08-04 08:51:51,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling ended: taskId = 456c1d7c-a0e4-4842-8e00-90faa50e28ef task status = finished
>2014-08-04 08:51:51,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling ended, spm status: SPM
>2014-08-04 08:51:51,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=456c1d7c-a0e4-4842-8e00-90faa50e28ef), log id: 64e69ea5
>2014-08-04 08:51:51,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] FINISH, HSMClearTaskVDSCommand, log id: 64e69ea5
>2014-08-04 08:51:51,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1835faa8, log id: 66cfa6c3
>2014-08-04 08:51:51,845 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 08:51:51,854 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 08:51:51,872 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 49bb32b6
>2014-08-04 08:51:51,904 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 08:51:51,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 68c432cc
>2014-08-04 08:51:51,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 68c432cc
>2014-08-04 08:51:51,912 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49bb32b6
>2014-08-04 08:51:51,913 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] Discovered no tasks on Storage Pool Default
>2014-08-04 09:23:55,969 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-04 09:24:24,422 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf.
>2014-08-04 10:23:55,973 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-08-04 10:24:24,438 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf.
>2014-08-04 11:23:55,979 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-04 11:24:24,454 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Attempting to update VMs/Templates Ovf.
>2014-08-04 11:51:33,310 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 11:51:48,437 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 11:53:10,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [78845858] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 11:53:10,168 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 11:53:10,172 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 11:53:10,174 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 11:53:10,191 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Correlation ID: 386bc8c5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 11:55:10,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 11:55:10,228 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [386bc8c5] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 11:55:10,228 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 11:55:10,229 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 11:55:10,256 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 11:55:10,268 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 21954c7d
>2014-08-04 11:55:10,270 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 77dc8f79
>2014-08-04 11:55:10,280 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 11:57:10,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 11:57:10,249 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [71be4699] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 11:57:10,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 11:57:10,258 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 11:58:10,281 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-08-04 11:58:10,283 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] FINISH, SpmStopVDSCommand, log id: 77dc8f79
>2014-08-04 11:58:10,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] FINISH, ResetIrsVDSCommand, log id: 21954c7d
>2014-08-04 11:58:10,298 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [343a7b93] Correlation ID: 343a7b93, Job ID: e798785d-fe15-45d8-91b4-164d267380eb, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 11:58:10,304 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [386bc8c5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 11:58:10,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 11:58:10,358 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 11:58:10,383 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 11:58:10,428 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Removing vds [] from the domain in maintenance cache
>2014-08-04 11:58:10,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 11:58:10,430 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [386bc8c5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 11:58:10,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 11:58:10,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 11:58:10,451 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 11:58:10,457 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 11:58:10,464 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 11:58:10,477 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Correlation ID: 37fc09d8, Job ID: f8c0d157-d357-498f-baf1-3fec31e86ac2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 11:58:10,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Removing vds [] from the domain in maintenance cache
>2014-08-04 11:58:10,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 11:59:10,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 11:59:10,225 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 11:59:10,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-04 11:59:10,996 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-08-04 11:59:11,011 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [aeb0141] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-04 11:59:11,019 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-04 11:59:14,256 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 21282722
>2014-08-04 11:59:14,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, GetHardwareInfoVDSCommand, log id: 21282722
>2014-08-04 11:59:14,338 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [6fb36f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 11:59:14,352 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [6fb36f9] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-04 11:59:14,356 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-37) [25c63078] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 11:59:14,364 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 11:59:14,370 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63b33708
>2014-08-04 11:59:14,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 63b33708
>2014-08-04 11:59:14,430 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] Host rack storage connection was succeeded 
>2014-08-04 11:59:14,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5bea21b
>2014-08-04 11:59:20,229 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 11:59:20,233 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 12:00:14,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 5bea21b
>2014-08-04 12:00:14,871 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1847737c
>2014-08-04 12:00:14,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1847737c
>2014-08-04 12:00:14,970 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [2c6893c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 12:00:14,976 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [2c6893c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 12:00:14,999 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [68f898a6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 12:00:20,342 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 12:00:20,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 12:00:20,351 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18f55877
>2014-08-04 12:00:20,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling started: taskId = 22eed789-2516-4fcd-97a0-0a74cd979b9c
>2014-08-04 12:00:21,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling ended: taskId = 22eed789-2516-4fcd-97a0-0a74cd979b9c task status = finished
>2014-08-04 12:00:21,465 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling ended, spm status: SPM
>2014-08-04 12:00:21,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=22eed789-2516-4fcd-97a0-0a74cd979b9c), log id: 15425cd5
>2014-08-04 12:00:21,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] FINISH, HSMClearTaskVDSCommand, log id: 15425cd5
>2014-08-04 12:00:21,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7965adf, log id: 18f55877
>2014-08-04 12:00:21,518 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 12:00:21,525 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 12:00:21,526 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 12:00:21,541 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 78f38c1a
>2014-08-04 12:00:21,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 12:00:21,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 315f0ec3
>2014-08-04 12:00:21,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 315f0ec3
>2014-08-04 12:00:21,574 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78f38c1a
>2014-08-04 12:00:21,575 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] Discovered no tasks on Storage Pool Default
>2014-08-04 12:23:55,986 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Principal admin::internal synchronized
>2014-08-04 12:24:24,469 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [14cac44e] Attempting to update VMs/Templates Ovf.
>2014-08-04 13:23:55,993 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-04 13:24:24,509 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
>2014-08-04 14:23:55,999 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-08-04 14:24:24,523 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Attempting to update VMs/Templates Ovf.
>2014-08-04 14:59:45,039 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-04 14:59:45,047 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 14:59:45,058 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [663a0e55] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 14:59:45,062 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [3f6766bd] Running command: LogoutUserCommand internal: false.
>2014-08-04 14:59:45,067 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3f6766bd] Correlation ID: 3f6766bd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 15:00:11,638 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 15:00:11,640 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 15:01:02,323 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-04 15:01:02,328 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 15:01:02,337 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [1299c980] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 15:01:02,342 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [6fe70835] Running command: LogoutUserCommand internal: false.
>2014-08-04 15:01:02,348 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [6fe70835] Correlation ID: 6fe70835, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 15:01:38,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 15:01:38,518 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 15:01:38,521 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [412fb852] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 15:01:38,523 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [412fb852] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 15:01:38,535 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [412fb852] Correlation ID: 412fb852, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 15:03:15,499 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-04 15:03:15,504 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 15:03:15,510 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [1835f83c] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 15:03:15,513 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [1f8a4f63] Running command: LogoutUserCommand internal: false.
>2014-08-04 15:03:15,517 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1f8a4f63] Correlation ID: 1f8a4f63, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 15:03:38,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 15:03:38,580 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [412fb852] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 15:03:38,584 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 15:03:38,587 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 15:03:38,625 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 15:03:38,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 668fe626
>2014-08-04 15:03:38,670 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 164dcfc0
>2014-08-04 15:03:38,688 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 15:04:55,393 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-04 15:04:55,397 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 15:04:55,404 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [22c856e0] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 15:04:55,418 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7f1f6a27] Running command: LogoutUserCommand internal: false.
>2014-08-04 15:04:55,423 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7f1f6a27] Correlation ID: 7f1f6a27, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 15:05:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Autorecovering 1 storage domains
>2014-08-04 15:05:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-04 15:05:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 15:05:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/4/14 3:05 PM
>2014-08-04 15:05:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-04 15:05:00,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5fde6970
>2014-08-04 15:05:38,538 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 15:05:38,541 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] FINISH, SpmStopVDSCommand, log id: 164dcfc0
>2014-08-04 15:05:38,545 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 15:05:38,548 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 15:05:38,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 15:05:38,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 15:05:38,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] FINISH, ConnectStorageServerVDSCommand, log id: 5fde6970
>2014-08-04 15:05:38,557 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] FINISH, ResetIrsVDSCommand, log id: 668fe626
>2014-08-04 15:05:38,556 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [42c2440e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 15:05:38,557 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-04 15:05:38,560 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [42c2440e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 15:05:38,572 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [473fd2de] Correlation ID: 473fd2de, Job ID: 43068271-5fdf-448f-bb9f-441b15d0aaab, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 15:05:38,564 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-04 15:05:38,593 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/4/14 3:05 PM
>2014-08-04 15:05:38,596 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [42c2440e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 15:05:38,599 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [412fb852] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 15:05:38,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 15:05:38,665 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 15:05:38,680 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 15:05:38,688 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 15:05:38,693 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 15:05:38,702 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Correlation ID: 3b604dc6, Job ID: 39312fcf-aa1f-4856-af2a-ccaf564ecc60, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 15:05:38,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Removing vds [] from the domain in maintenance cache
>2014-08-04 15:05:38,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 15:05:38,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 15:05:38,715 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 15:05:38,731 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 15:05:38,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] Removing vds [] from the domain in maintenance cache
>2014-08-04 15:05:38,738 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 15:05:38,739 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [412fb852] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 15:05:41,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 155034f3
>2014-08-04 15:05:41,929 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, GetHardwareInfoVDSCommand, log id: 155034f3
>2014-08-04 15:05:41,971 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [56bacfd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 15:05:41,981 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-74) [458b4be2] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 15:05:41,989 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 15:05:42,000 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40e5cf30
>2014-08-04 15:05:42,063 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 40e5cf30
>2014-08-04 15:05:42,064 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] Host rack storage connection was succeeded 
>2014-08-04 15:05:42,073 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7b101cd7
>2014-08-04 15:05:48,555 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-29) [327b509d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 15:05:48,558 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-29) [327b509d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 15:06:42,468 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, ConnectStoragePoolVDSCommand, log id: 7b101cd7
>2014-08-04 15:06:42,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 39925c2c
>2014-08-04 15:06:42,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 39925c2c
>2014-08-04 15:06:42,580 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [32674fb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 15:06:42,587 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [32674fb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 15:06:42,603 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [5972dc2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 15:06:48,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 15:06:48,723 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 15:06:48,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46b44774
>2014-08-04 15:06:48,774 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling started: taskId = a43d14d6-dac0-44ef-a3a9-588e590388a4
>2014-08-04 15:06:49,781 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling ended: taskId = a43d14d6-dac0-44ef-a3a9-588e590388a4 task status = finished
>2014-08-04 15:06:49,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling ended, spm status: SPM
>2014-08-04 15:06:49,840 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=a43d14d6-dac0-44ef-a3a9-588e590388a4), log id: 4435c065
>2014-08-04 15:06:49,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] FINISH, HSMClearTaskVDSCommand, log id: 4435c065
>2014-08-04 15:06:49,886 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e88223d, log id: 46b44774
>2014-08-04 15:06:49,891 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 15:06:49,899 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 15:06:49,901 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 15:06:49,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 22699016
>2014-08-04 15:06:49,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 15:06:49,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 33a670c
>2014-08-04 15:06:49,950 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33a670c
>2014-08-04 15:06:49,951 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22699016
>2014-08-04 15:06:49,952 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] Discovered no tasks on Storage Pool Default
>2014-08-04 15:23:56,006 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-08-04 15:24:24,543 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf.
>2014-08-04 16:00:59,087 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-04 16:00:59,092 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:00:59,100 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [74f9591] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:00:59,102 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [2249ac9f] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:00:59,107 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2249ac9f] Correlation ID: 2249ac9f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:01:53,134 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-04 16:01:53,140 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:01:53,146 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [78671431] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:01:53,148 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [54cb46ae] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:01:53,152 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [54cb46ae] Correlation ID: 54cb46ae, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:03:44,994 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-04 16:03:44,998 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:03:45,006 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [1a52fe7c] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:03:45,010 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [20d8892c] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:03:45,015 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [20d8892c] Correlation ID: 20d8892c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:09:21,502 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-04 16:09:21,508 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:09:21,518 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [127e1cff] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:09:21,523 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [1a57a014] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:09:21,529 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1a57a014] Correlation ID: 1a57a014, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:15:03,561 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-04 16:15:03,571 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:15:03,578 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [2b0e536a] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:15:03,581 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [10322835] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:15:03,585 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [10322835] Correlation ID: 10322835, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:23:56,011 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Principal admin::internal synchronized
>2014-08-04 16:24:24,557 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [aa1055d] Attempting to update VMs/Templates Ovf.
>2014-08-04 16:30:46,035 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-04 16:30:46,053 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:30:46,059 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7e1ba738] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:30:46,062 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [581cece6] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:30:46,066 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [581cece6] Correlation ID: 581cece6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 16:33:09,626 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-04 16:33:09,631 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-04 16:33:09,637 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [43e1cb92] Running command: LogoutBySessionCommand internal: false.
>2014-08-04 16:33:09,639 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [6e7a474d] Running command: LogoutUserCommand internal: false.
>2014-08-04 16:33:09,643 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6e7a474d] Correlation ID: 6e7a474d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-04 17:20:09,784 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-33) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 17:23:56,016 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Principal admin::internal synchronized
>2014-08-04 17:24:24,571 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf.
>2014-08-04 17:39:46,542 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-33) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 18:06:35,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 18:06:35,731 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 18:08:06,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 18:08:06,332 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 18:08:06,336 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 18:08:06,339 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 18:08:06,354 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Correlation ID: 1ecc4e7f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 18:10:06,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 18:10:06,381 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 18:10:06,381 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 18:10:06,382 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 18:10:06,403 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 18:10:06,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4caf530b
>2014-08-04 18:10:06,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 249b0ca4
>2014-08-04 18:10:06,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 18:12:06,354 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 18:12:06,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] FINISH, SpmStopVDSCommand, log id: 249b0ca4
>2014-08-04 18:12:06,359 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [5972dc2] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 18:12:06,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-74) [5972dc2] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 18:12:06,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 18:12:06,364 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 18:12:06,365 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 18:12:06,372 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] FINISH, ResetIrsVDSCommand, log id: 4caf530b
>2014-08-04 18:12:06,397 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 18:12:06,400 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] Correlation ID: 35e84c4a, Job ID: 96d93662-6798-4e77-a3cc-2c3aebc9afeb, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 18:12:06,404 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [1ecc4e7f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 18:12:06,406 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 18:12:06,457 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 18:12:06,474 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-49) [67143c69] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 18:12:06,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [67143c69] Removing vds [] from the domain in maintenance cache
>2014-08-04 18:12:06,482 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [67143c69] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 18:12:06,483 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [1ecc4e7f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 18:12:06,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 18:12:06,489 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 18:12:06,504 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 18:12:06,510 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 18:12:06,515 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 18:12:06,525 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Correlation ID: 26119ccb, Job ID: 6103c3b6-ac84-41a3-8cf9-2d27d69a53f3, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 18:12:06,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Removing vds [] from the domain in maintenance cache
>2014-08-04 18:12:06,532 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 18:12:09,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: f4310ef
>2014-08-04 18:12:09,727 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, GetHardwareInfoVDSCommand, log id: f4310ef
>2014-08-04 18:12:09,752 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [f66741c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 18:12:09,801 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-21) [41c4a42f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 18:12:09,809 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 18:12:09,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42f51dff
>2014-08-04 18:12:09,872 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 42f51dff
>2014-08-04 18:12:09,873 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] Host rack storage connection was succeeded 
>2014-08-04 18:12:09,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2117c1e9
>2014-08-04 18:12:16,371 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 18:12:16,375 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 18:13:10,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, ConnectStoragePoolVDSCommand, log id: 2117c1e9
>2014-08-04 18:13:10,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45bc50ff
>2014-08-04 18:13:10,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45bc50ff
>2014-08-04 18:13:10,389 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [1ccec212] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 18:13:10,395 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [1ccec212] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 18:13:10,411 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [675f1edc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 18:13:16,516 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 18:13:16,522 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 18:13:16,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78b5e37d
>2014-08-04 18:13:16,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling started: taskId = ee383937-c5ce-4a40-bbfe-c58e7ec18c06
>2014-08-04 18:13:17,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended: taskId = ee383937-c5ce-4a40-bbfe-c58e7ec18c06 task status = finished
>2014-08-04 18:13:17,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended, spm status: SPM
>2014-08-04 18:13:17,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=ee383937-c5ce-4a40-bbfe-c58e7ec18c06), log id: 2ab3b892
>2014-08-04 18:13:17,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: 2ab3b892
>2014-08-04 18:13:17,681 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14f2f5c8, log id: 78b5e37d
>2014-08-04 18:13:17,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 18:13:17,692 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 18:13:17,695 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 18:13:17,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f5984ea
>2014-08-04 18:13:17,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 18:13:17,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1b18ff31
>2014-08-04 18:13:17,740 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b18ff31
>2014-08-04 18:13:17,741 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f5984ea
>2014-08-04 18:13:17,742 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) Discovered no tasks on Storage Pool Default
>2014-08-04 18:23:56,022 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized
>2014-08-04 18:24:24,606 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf.
>2014-08-04 19:17:44,691 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 19:23:56,027 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized
>2014-08-04 19:24:24,623 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) [52d379dc] Attempting to update VMs/Templates Ovf.
>2014-08-04 20:23:56,033 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) [6452f878] Principal admin::internal synchronized
>2014-08-04 20:24:24,641 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf.
>2014-08-04 21:12:55,259 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-04 21:13:10,409 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-04 21:14:31,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 21:14:31,551 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 21:14:31,557 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [4e09115c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-04 21:14:31,562 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [4e09115c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-04 21:14:31,577 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [4e09115c] Correlation ID: 4e09115c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-04 21:16:31,595 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 21:16:31,597 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [4e09115c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 21:16:31,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-04 21:16:31,599 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-04 21:16:31,643 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-04 21:16:31,663 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 77d7eddf
>2014-08-04 21:16:31,667 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3435d519
>2014-08-04 21:16:31,673 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-04 21:18:31,567 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-04 21:18:31,570 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 21:18:31,571 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-04 21:18:31,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-04 21:18:31,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] FINISH, SpmStopVDSCommand, log id: 3435d519
>2014-08-04 21:18:31,574 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-04 21:18:31,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-04 21:18:31,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] FINISH, ResetIrsVDSCommand, log id: 77d7eddf
>2014-08-04 21:18:31,581 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-04 21:18:31,591 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] Correlation ID: 5e32ac81, Job ID: a6f6dadd-f506-4c1b-8739-9fc8e47a5f3b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-04 21:18:31,614 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-04 21:18:31,615 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [4e09115c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 21:18:31,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-04 21:18:31,683 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-04 21:18:31,702 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-45) [efb44de] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-04 21:18:31,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [efb44de] Removing vds [] from the domain in maintenance cache
>2014-08-04 21:18:31,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [efb44de] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 21:18:31,711 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [4e09115c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-04 21:18:31,713 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-04 21:18:31,718 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-04 21:18:31,733 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 21:18:31,740 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-04 21:18:31,745 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-04 21:18:31,754 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [44482845] Correlation ID: 44482845, Job ID: 80fcfbd9-a756-47d5-a32a-d0b39d9522b8, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-04 21:18:31,759 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [44482845] Removing vds [] from the domain in maintenance cache
>2014-08-04 21:18:31,761 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [44482845] Removing host(s) [] from hosts unseen domain report cache
>2014-08-04 21:18:34,866 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3dff2683
>2014-08-04 21:18:34,914 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] FINISH, GetHardwareInfoVDSCommand, log id: 3dff2683
>2014-08-04 21:18:34,938 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [a48784f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 21:18:34,988 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [58cb2ded] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 21:18:34,996 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-04 21:18:35,011 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42dae9a8
>2014-08-04 21:18:35,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 42dae9a8
>2014-08-04 21:18:35,078 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] Host rack storage connection was succeeded 
>2014-08-04 21:18:35,087 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 49dbab5f
>2014-08-04 21:18:41,580 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 21:18:41,584 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-04 21:19:35,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 49dbab5f
>2014-08-04 21:19:35,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 168b56d0
>2014-08-04 21:19:35,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] FINISH, SetMOMPolicyParametersVDSCommand, log id: 168b56d0
>2014-08-04 21:19:35,654 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [73399a21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-04 21:19:35,659 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [73399a21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-04 21:19:35,674 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [48fc48e5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-04 21:19:41,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-04 21:19:41,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-04 21:19:41,717 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b609664
>2014-08-04 21:19:41,765 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling started: taskId = 7ba16e70-2fa1-4939-80c7-bef94b275b97
>2014-08-04 21:19:42,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling ended: taskId = 7ba16e70-2fa1-4939-80c7-bef94b275b97 task status = finished
>2014-08-04 21:19:42,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling ended, spm status: SPM
>2014-08-04 21:19:42,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-88) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ba16e70-2fa1-4939-80c7-bef94b275b97), log id: 6ce4ec85
>2014-08-04 21:19:42,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, HSMClearTaskVDSCommand, log id: 6ce4ec85
>2014-08-04 21:19:42,876 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@269fcb0e, log id: 1b609664
>2014-08-04 21:19:42,880 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-04 21:19:42,887 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-04 21:19:42,889 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-04 21:19:42,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 75655cc8
>2014-08-04 21:19:42,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-04 21:19:42,937 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2b804f20
>2014-08-04 21:19:42,942 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b804f20
>2014-08-04 21:19:42,943 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75655cc8
>2014-08-04 21:19:42,944 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool Default
>2014-08-04 21:23:56,040 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Principal admin::internal synchronized
>2014-08-04 21:24:24,655 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [4341002] Attempting to update VMs/Templates Ovf.
>2014-08-04 22:23:56,051 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized
>2014-08-04 22:24:24,668 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf.
>2014-08-04 23:23:56,058 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Principal admin::internal synchronized
>2014-08-04 23:24:24,701 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
>2014-08-05 00:19:30,599 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 00:19:30,601 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 00:20:59,983 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [68f898a6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 00:20:59,988 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 00:20:59,992 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-37) [10578196] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 00:20:59,994 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-37) [10578196] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 00:21:00,008 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [10578196] Correlation ID: 10578196, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 00:23:00,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 00:23:00,039 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [10578196] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 00:23:00,040 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-37) [10578196] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 00:23:00,041 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 00:23:00,072 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 00:23:00,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 47eb89c9
>2014-08-05 00:23:00,100 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6d337b4f
>2014-08-05 00:23:00,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 00:23:56,063 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Principal admin::internal synchronized
>2014-08-05 00:24:24,713 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) [65d9de58] Attempting to update VMs/Templates Ovf.
>2014-08-05 00:25:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Autorecovering 1 storage domains
>2014-08-05 00:25:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-05 00:25:00,012 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 00:25:00,010 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 00:25:00,020 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 00:25:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 00:25:00,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 00:25:00,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 00:25:00,026 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 12:25 AM
>2014-08-05 00:25:00,023 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 00:25:00,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] FINISH, SpmStopVDSCommand, log id: 6d337b4f
>2014-08-05 00:25:00,036 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-05 00:25:00,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15d9f1b0
>2014-08-05 00:25:00,058 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 00:25:00,057 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] FINISH, ResetIrsVDSCommand, log id: 47eb89c9
>2014-08-05 00:25:00,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 00:25:00,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] FINISH, ConnectStorageServerVDSCommand, log id: 15d9f1b0
>2014-08-05 00:25:00,068 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-05 00:25:00,071 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-05 00:25:00,072 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 12:25 AM
>2014-08-05 00:25:00,076 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] Correlation ID: 2bbb80ac, Job ID: 36c08b8e-d5ed-4e31-8f15-324c5c6d989c, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 00:25:00,085 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-23) [10578196] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 00:25:00,088 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 00:25:00,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 00:25:00,175 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 00:25:00,182 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 00:25:00,188 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 00:25:00,198 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Correlation ID: 4e787b0d, Job ID: 970b1f0b-ef60-4c89-b93a-4d6589187afd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 00:25:00,203 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Removing vds [] from the domain in maintenance cache
>2014-08-05 00:25:00,204 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 00:25:00,206 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 00:25:00,212 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 00:25:00,230 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 00:25:00,237 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] Removing vds [] from the domain in maintenance cache
>2014-08-05 00:25:00,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 00:25:00,239 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-23) [10578196] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 00:25:03,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78cc51f3
>2014-08-05 00:25:03,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] HostName = rack
>2014-08-05 00:25:03,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 00:25:03,149 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-05 00:25:03,153 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-05 00:25:06,386 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 720e86
>2014-08-05 00:25:06,436 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 720e86
>2014-08-05 00:25:06,453 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [439dfcd4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 00:25:06,462 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [439dfcd4] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-05 00:25:06,463 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [7eb8d6ee] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 00:25:06,470 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 00:25:06,481 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2d32f3a7
>2014-08-05 00:25:06,537 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 2d32f3a7
>2014-08-05 00:25:06,538 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] Host rack storage connection was succeeded 
>2014-08-05 00:25:06,543 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 652696f9
>2014-08-05 00:25:10,030 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) [58bfe349] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 00:25:10,034 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) [58bfe349] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 00:26:06,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 652696f9
>2014-08-05 00:26:06,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 46b6558f
>2014-08-05 00:26:07,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] FINISH, SetMOMPolicyParametersVDSCommand, log id: 46b6558f
>2014-08-05 00:26:07,047 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [55f57cce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 00:26:07,055 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [55f57cce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 00:26:07,078 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [7dde64c0] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 00:26:10,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 00:26:10,175 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 00:26:10,177 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f76d94b
>2014-08-05 00:26:10,226 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling started: taskId = 00e33615-bd15-4363-acff-3b0446834773
>2014-08-05 00:26:11,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling ended: taskId = 00e33615-bd15-4363-acff-3b0446834773 task status = finished
>2014-08-05 00:26:11,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling ended, spm status: SPM
>2014-08-05 00:26:11,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=00e33615-bd15-4363-acff-3b0446834773), log id: 2d9d319e
>2014-08-05 00:26:11,339 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] FINISH, HSMClearTaskVDSCommand, log id: 2d9d319e
>2014-08-05 00:26:11,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4fbf4b05, log id: 6f76d94b
>2014-08-05 00:26:11,344 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 00:26:11,350 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [519c108f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 00:26:11,361 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 34abe4f1
>2014-08-05 00:26:11,389 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 00:26:11,391 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 55e32341
>2014-08-05 00:26:11,397 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e32341
>2014-08-05 00:26:11,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34abe4f1
>2014-08-05 00:26:11,399 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) [519c108f] Discovered no tasks on Storage Pool Default
>2014-08-05 01:23:56,069 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Principal admin::internal synchronized
>2014-08-05 01:24:24,715 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Attempting to update VMs/Templates Ovf.
>2014-08-05 02:23:56,078 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized
>2014-08-05 02:24:24,732 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf.
>2014-08-05 03:23:56,084 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized
>2014-08-05 03:24:24,751 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf.
>2014-08-05 03:26:04,814 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 03:26:04,816 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 03:27:30,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 03:27:30,366 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 03:27:30,370 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-90) [12692311] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 03:27:30,372 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-90) [12692311] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 03:27:30,383 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [12692311] Correlation ID: 12692311, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 03:29:30,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 03:29:30,429 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) [12692311] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 03:29:30,430 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [12692311] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 03:29:30,431 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 03:29:30,475 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 03:29:30,498 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 198fd227
>2014-08-05 03:29:30,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 66bea16c
>2014-08-05 03:29:30,547 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 03:30:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) Autorecovering 1 storage domains
>2014-08-05 03:30:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-05 03:30:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 03:30:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 3:30 AM
>2014-08-05 03:30:00,018 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-05 03:30:00,023 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 562929a4
>2014-08-05 03:31:30,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 03:31:30,409 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 03:31:30,410 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 03:31:30,411 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 03:32:30,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 562929a4
>2014-08-05 03:32:30,505 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 3:32 AM
>2014-08-05 03:32:30,548 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-08-05 03:32:30,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] FINISH, SpmStopVDSCommand, log id: 66bea16c
>2014-08-05 03:32:30,556 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] FINISH, ResetIrsVDSCommand, log id: 198fd227
>2014-08-05 03:32:30,566 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [6d21d912] Correlation ID: 6d21d912, Job ID: f5df7420-b8d2-4b9c-aa1f-913622b83139, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 03:32:30,573 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [12692311] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 03:32:30,575 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 03:32:30,637 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 03:32:30,660 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 03:32:30,684 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Removing vds [] from the domain in maintenance cache
>2014-08-05 03:32:30,685 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 03:32:30,686 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [12692311] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 03:32:30,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 03:32:30,693 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 03:32:30,710 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 03:32:30,716 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 03:32:30,728 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 03:32:30,742 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Correlation ID: 47b01ade, Job ID: ed978873-7309-4d0f-a618-a58b6deff278, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 03:32:30,747 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Removing vds [] from the domain in maintenance cache
>2014-08-05 03:32:30,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 03:33:30,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 03:33:30,426 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 03:33:31,177 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 03:33:31,179 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 03:33:31,191 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-05 03:33:31,199 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-05 03:33:34,432 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4fdd9f51
>2014-08-05 03:33:34,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GetHardwareInfoVDSCommand, log id: 4fdd9f51
>2014-08-05 03:33:34,511 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [3f9d9215] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 03:33:34,561 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [3f9d9215] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-05 03:33:34,563 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-96) [5368d9dd] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 03:33:34,571 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 03:33:34,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6030bbc6
>2014-08-05 03:33:34,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6030bbc6
>2014-08-05 03:33:34,640 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] Host rack storage connection was succeeded 
>2014-08-05 03:33:34,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 85d57cb
>2014-08-05 03:33:40,430 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 03:33:40,434 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 03:34:35,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 85d57cb
>2014-08-05 03:34:35,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 125596
>2014-08-05 03:34:35,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 125596
>2014-08-05 03:34:35,169 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [6941298b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 03:34:35,176 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [6941298b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 03:34:35,194 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [3648c2dc] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 03:34:40,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 03:34:40,551 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 03:34:40,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62c89002
>2014-08-05 03:34:40,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = 6a20d58f-0e63-4969-b02c-d9f94fcd3f67
>2014-08-05 03:34:41,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = 6a20d58f-0e63-4969-b02c-d9f94fcd3f67 task status = finished
>2014-08-05 03:34:41,662 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM
>2014-08-05 03:34:41,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6a20d58f-0e63-4969-b02c-d9f94fcd3f67), log id: 65f5ada6
>2014-08-05 03:34:41,710 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 65f5ada6
>2014-08-05 03:34:41,712 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f93cf03, log id: 62c89002
>2014-08-05 03:34:41,716 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 03:34:41,724 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 03:34:41,726 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 03:34:41,742 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4b74362b
>2014-08-05 03:34:41,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 03:34:41,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e792ca4
>2014-08-05 03:34:41,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e792ca4
>2014-08-05 03:34:41,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4b74362b
>2014-08-05 03:34:41,792 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [4760e42] Discovered no tasks on Storage Pool Default
>2014-08-05 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-81) [7125e62c] Start deleteAgedOutAuditLogs
>2014-08-05 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Start deleteAgedOutCommandEntities
>2014-08-05 03:35:35,007 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-81) [7125e62c] Finished deleteAgedOutAuditLogs
>2014-08-05 03:35:35,009 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Finished deleteAgedOutCommandEntities
>2014-08-05 04:23:56,089 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized
>2014-08-05 04:24:24,772 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
>2014-08-05 05:23:56,095 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized
>2014-08-05 05:24:24,807 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) [4760e42] Attempting to update VMs/Templates Ovf.
>2014-08-05 06:23:56,102 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Principal admin::internal synchronized
>2014-08-05 06:24:24,843 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf.
>2014-08-05 06:34:20,632 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 06:34:20,634 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 06:35:56,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 06:35:56,591 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 06:35:56,595 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 06:35:56,598 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 06:35:56,613 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Correlation ID: 5b2cf9d8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 06:37:56,643 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 06:37:56,646 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 06:37:56,647 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 06:37:56,648 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 06:37:56,693 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 06:37:56,715 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 678172ed
>2014-08-05 06:37:56,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2faa05d8
>2014-08-05 06:37:56,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 06:39:56,613 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 06:39:56,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 06:39:56,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 06:39:56,616 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] FINISH, SpmStopVDSCommand, log id: 2faa05d8
>2014-08-05 06:39:56,621 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [14ab6114] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 06:39:56,623 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [14ab6114] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 06:39:56,620 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 06:39:56,624 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [14ab6114] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 06:39:56,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] FINISH, ResetIrsVDSCommand, log id: 678172ed
>2014-08-05 06:39:56,656 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [14ab6114] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 06:39:56,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [54d106fb] Correlation ID: 54d106fb, Job ID: 972c6800-188a-4a8b-b123-7aa1d39d00a5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 06:39:56,674 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-12) [5b2cf9d8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 06:39:56,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 06:39:56,763 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 06:39:56,780 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [7043d56e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 06:39:56,787 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [7043d56e] Removing vds [] from the domain in maintenance cache
>2014-08-05 06:39:56,788 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [7043d56e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 06:39:56,789 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-12) [5b2cf9d8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 06:39:56,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 06:39:56,796 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 06:39:56,811 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 06:39:56,818 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 06:39:56,823 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 06:39:56,833 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Correlation ID: 6bbc9db9, Job ID: ef20d657-e1e2-42a3-a8e8-2540f7918007, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 06:39:56,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Removing vds [] from the domain in maintenance cache
>2014-08-05 06:39:56,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 06:39:59,733 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3e9f5f42
>2014-08-05 06:39:59,734 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] HostName = rack
>2014-08-05 06:39:59,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 06:39:59,748 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-05 06:39:59,756 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [782c75f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-05 06:40:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Autorecovering 2 storage domains
>2014-08-05 06:40:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-05 06:40:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 06:40:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 6:40 AM
>2014-08-05 06:40:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 6:40 AM
>2014-08-05 06:40:00,018 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [150723da] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-08-05 06:40:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 06:40:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 6:40 AM
>2014-08-05 06:40:00,025 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 6:40 AM
>2014-08-05 06:40:02,991 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6e5bc94a
>2014-08-05 06:40:03,041 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] FINISH, GetHardwareInfoVDSCommand, log id: 6e5bc94a
>2014-08-05 06:40:03,058 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [31903ec5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 06:40:03,104 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) [31903ec5] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-05 06:40:03,105 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [7ebb35ee] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 06:40:03,111 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 06:40:03,117 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c92140a
>2014-08-05 06:40:03,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4c92140a
>2014-08-05 06:40:03,177 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] Host rack storage connection was succeeded 
>2014-08-05 06:40:03,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7162ee48
>2014-08-05 06:40:06,628 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 06:40:06,633 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 06:41:03,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 7162ee48
>2014-08-05 06:41:03,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 347ae785
>2014-08-05 06:41:03,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] FINISH, SetMOMPolicyParametersVDSCommand, log id: 347ae785
>2014-08-05 06:41:03,723 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [4f16934] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 06:41:03,731 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [4f16934] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 06:41:03,751 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 06:41:06,748 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 06:41:06,755 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 06:41:06,759 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74393759
>2014-08-05 06:41:06,810 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling started: taskId = 56799788-dd0e-46b4-955b-3a4cd8bdb866
>2014-08-05 06:41:07,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling ended: taskId = 56799788-dd0e-46b4-955b-3a4cd8bdb866 task status = finished
>2014-08-05 06:41:07,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling ended, spm status: SPM
>2014-08-05 06:41:07,879 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=56799788-dd0e-46b4-955b-3a4cd8bdb866), log id: 1b7df594
>2014-08-05 06:41:07,924 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] FINISH, HSMClearTaskVDSCommand, log id: 1b7df594
>2014-08-05 06:41:07,925 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@206d3049, log id: 74393759
>2014-08-05 06:41:07,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 06:41:07,936 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 06:41:07,938 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 06:41:07,949 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7761496
>2014-08-05 06:41:07,975 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 06:41:07,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 291e2a13
>2014-08-05 06:41:07,982 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 291e2a13
>2014-08-05 06:41:07,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7761496
>2014-08-05 06:41:07,985 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] Discovered no tasks on Storage Pool Default
>2014-08-05 06:54:52,529 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-37) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 07:15:59,741 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-8) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 07:23:56,107 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Principal admin::internal synchronized
>2014-08-05 07:24:24,877 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [1f9421b1] Attempting to update VMs/Templates Ovf.
>2014-08-05 08:23:56,115 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) [4760e42] Principal admin::internal synchronized
>2014-08-05 08:24:24,899 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Attempting to update VMs/Templates Ovf.
>2014-08-05 09:23:56,125 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized
>2014-08-05 09:24:24,913 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) [48cf3129] Attempting to update VMs/Templates Ovf.
>2014-08-05 09:40:55,390 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 09:40:55,392 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 09:42:27,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [7dde64c0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 09:42:27,402 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 09:42:27,406 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 09:42:27,408 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 09:42:27,420 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Correlation ID: 5b5ccf8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 09:44:27,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 09:44:27,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 09:44:27,464 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 09:44:27,465 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 09:44:27,508 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 09:44:27,529 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 79a4d674
>2014-08-05 09:44:27,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4ee75edb
>2014-08-05 09:44:27,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 09:45:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [4341002] Autorecovering 1 storage domains
>2014-08-05 09:45:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [4341002] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-05 09:45:00,007 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 09:45:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 9:45 AM
>2014-08-05 09:45:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-05 09:45:00,025 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4a6166bf
>2014-08-05 09:46:27,431 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 09:46:27,443 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 09:46:27,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] FINISH, SpmStopVDSCommand, log id: 4ee75edb
>2014-08-05 09:46:27,442 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-83) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 09:46:27,441 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 09:46:27,464 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 09:46:27,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] FINISH, ResetIrsVDSCommand, log id: 79a4d674
>2014-08-05 09:46:27,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 09:46:27,475 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 09:46:27,489 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] Correlation ID: 2897c8c0, Job ID: edacbacd-2001-482d-b262-7b2810e9c972, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 09:46:27,480 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] FINISH, ConnectStorageServerVDSCommand, log id: 4a6166bf
>2014-08-05 09:46:27,511 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-05 09:46:27,523 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-05 09:46:27,525 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 9:46 AM
>2014-08-05 09:46:27,526 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-22) [5b5ccf8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 09:46:27,528 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 09:46:27,538 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 09:46:27,590 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 09:46:27,606 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 09:46:27,613 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 09:46:27,618 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 09:46:27,626 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Correlation ID: 18dc9b9c, Job ID: b0afea17-65c6-41e8-a7db-8287a9099f43, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 09:46:27,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Removing vds [] from the domain in maintenance cache
>2014-08-05 09:46:27,632 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 09:46:27,634 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 09:46:27,639 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 09:46:27,662 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [4340f131] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 09:46:27,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [4340f131] Removing vds [] from the domain in maintenance cache
>2014-08-05 09:46:27,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [4340f131] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 09:46:27,670 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-22) [5b5ccf8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 09:46:30,704 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1fb31abe
>2014-08-05 09:46:30,706 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) HostName = rack
>2014-08-05 09:46:30,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 09:46:30,726 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-05 09:46:30,734 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-05 09:46:33,984 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 363f0058
>2014-08-05 09:46:34,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] FINISH, GetHardwareInfoVDSCommand, log id: 363f0058
>2014-08-05 09:46:34,090 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [4a9cb1ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 09:46:34,100 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) [4a9cb1ef] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-05 09:46:34,102 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [655d9eeb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 09:46:34,116 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 09:46:34,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70acfc72
>2014-08-05 09:46:34,164 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 70acfc72
>2014-08-05 09:46:34,165 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Host rack storage connection was succeeded 
>2014-08-05 09:46:34,172 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 232fb377
>2014-08-05 09:46:37,487 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-91) [64566e87] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 09:46:37,492 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-91) [64566e87] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 09:47:34,568 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 232fb377
>2014-08-05 09:47:34,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3d7ac75b
>2014-08-05 09:47:34,679 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3d7ac75b
>2014-08-05 09:47:34,689 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 09:47:34,697 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 09:47:34,725 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [10e8de77] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 09:47:37,688 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 09:47:37,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 09:47:37,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c6254d8
>2014-08-05 09:47:37,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling started: taskId = 01c1a364-29b3-4511-bc23-3ab6d412a182
>2014-08-05 09:47:38,756 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling ended: taskId = 01c1a364-29b3-4511-bc23-3ab6d412a182 task status = finished
>2014-08-05 09:47:38,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling ended, spm status: SPM
>2014-08-05 09:47:38,815 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=01c1a364-29b3-4511-bc23-3ab6d412a182), log id: 43382406
>2014-08-05 09:47:38,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, HSMClearTaskVDSCommand, log id: 43382406
>2014-08-05 09:47:38,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bf8b515, log id: 6c6254d8
>2014-08-05 09:47:38,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 09:47:38,872 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 09:47:38,873 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 09:47:38,884 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: de30a73
>2014-08-05 09:47:38,911 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 09:47:38,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3131d937
>2014-08-05 09:47:38,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3131d937
>2014-08-05 09:47:38,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: de30a73
>2014-08-05 09:47:38,920 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-10) Discovered no tasks on Storage Pool Default
>2014-08-05 10:23:56,134 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) [48cf3129] Principal admin::internal synchronized
>2014-08-05 10:24:24,928 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [1b83a377] Attempting to update VMs/Templates Ovf.
>2014-08-05 10:58:07,282 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-05 10:58:07,290 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-05 10:58:07,306 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [66c1e4ac] Running command: LogoutBySessionCommand internal: false.
>2014-08-05 10:58:07,311 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [44a1e617] Running command: LogoutUserCommand internal: false.
>2014-08-05 10:58:07,318 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [44a1e617] Correlation ID: 44a1e617, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-05 10:58:35,535 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-05 10:58:35,539 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-05 10:58:35,549 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [5119ae73] Running command: LogoutBySessionCommand internal: false.
>2014-08-05 10:58:35,552 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [4a7eb4b7] Running command: LogoutUserCommand internal: false.
>2014-08-05 10:58:35,556 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [4a7eb4b7] Correlation ID: 4a7eb4b7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-05 10:58:40,989 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-05 10:58:40,993 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-05 10:58:41,463 INFO  [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-1) [50420e3c-b64b-4342_addDataCenter_create] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-05 10:58:41,491 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [50420e3c-b64b-4342_addDataCenter_create] Correlation ID: 50420e3c-b64b-4342_addDataCenter_create, Call Stack: null, Custom Event ID: -1, Message: Data Center datacenter_None, Compatibility Version 3.4 and Quota Type DISABLED was added by admin
>2014-08-05 10:58:43,788 INFO  [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-4) [bcf22733-13b6-486e_addCluster_create] Running command: AddVdsGroupCommand internal: false. Entities affected :  ID: b69d184e-0a37-4ed1-b097-86e0d2968149 Type: StoragePool
>2014-08-05 10:58:43,854 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [bcf22733-13b6-486e_addCluster_create] Correlation ID: bcf22733-13b6-486e_addCluster_create, Call Stack: null, Custom Event ID: -1, Message: Cluster cluster_None was added by admin
>2014-08-05 10:58:50,990 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-10) [c8d39794-ba0b-4cf5_addHost_create] Failed to authenticate session with host slot-7.rhev.lab.eng.brq.redhat.com: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.63.137' failed. Please verify provided credentials. Make sure host is configured for password authentication
>	at org.ovirt.engine.core.uutils.ssh.SSHClient.authenticate(SSHClient.java:365) [uutils.jar:]
>	at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:430) [bll.jar:]
>	at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:342) [bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:737) [bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:339) [bll.jar:]
>	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:]
>	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:]
>	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:369) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
>	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
>	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view7.runAction(Unknown Source) [common.jar:]
>	at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:259)
>	at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243)
>	at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120)
>	at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110)
>	at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129)
>	at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100)
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_55]
>	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
>	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:]
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110)
>	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101)
>	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
>	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
>	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
>	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
>	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
>	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
>	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
>	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
>	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
>	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
>	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
>	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 10:58:51,100 WARN  [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-10) [c8d39794-ba0b-4cf5_addHost_create] CanDoAction of action AddVds failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__HOST,$server 10.34.63.137,VDS_CANNOT_AUTHENTICATE_TO_SERVER
>2014-08-05 10:58:51,119 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [Cannot add Host. SSH authentication failed, verify authentication parameters are correct (Username/Password, public-key etc.) You may refer to the engine.log file for further details.]
>2014-08-05 11:23:56,140 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized
>2014-08-05 11:24:24,967 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf.
>2014-08-05 12:23:56,150 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Principal admin::internal synchronized
>2014-08-05 12:24:24,996 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
>2014-08-05 12:47:18,129 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 12:47:33,260 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 12:48:58,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 12:48:58,031 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 12:48:58,035 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 12:48:58,037 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 12:48:58,052 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Correlation ID: 5d663c2e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 12:50:58,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 12:50:58,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [5d663c2e] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 12:50:58,080 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 12:50:58,081 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 12:50:58,117 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 12:50:58,136 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5eba1a6f
>2014-08-05 12:50:58,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 45b2da5a
>2014-08-05 12:50:58,146 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 12:52:58,047 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 12:52:58,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 12:52:58,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-63) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 12:52:58,054 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] FINISH, SpmStopVDSCommand, log id: 45b2da5a
>2014-08-05 12:52:58,058 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 12:52:58,058 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 12:52:58,065 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] FINISH, ResetIrsVDSCommand, log id: 5eba1a6f
>2014-08-05 12:52:58,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 12:52:58,081 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [66d0e553] Correlation ID: 66d0e553, Job ID: 7d1d8dac-1268-418e-b3a0-6b5ff20b3f54, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 12:52:58,126 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [5d663c2e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 12:52:58,130 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 12:52:58,131 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 12:52:58,185 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 12:52:58,202 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [fe90986] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 12:52:58,209 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [fe90986] Removing vds [] from the domain in maintenance cache
>2014-08-05 12:52:58,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [fe90986] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 12:52:58,211 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [5d663c2e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 12:52:58,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 12:52:58,217 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 12:52:58,231 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 12:52:58,238 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 12:52:58,243 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 12:52:58,254 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Correlation ID: 36e57e40, Job ID: f6e6aa62-cc55-43a0-91e7-c8f1b1ba6157, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 12:52:58,259 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Removing vds [] from the domain in maintenance cache
>2014-08-05 12:52:58,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 12:53:01,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: be2058e
>2014-08-05 12:53:01,456 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] FINISH, GetHardwareInfoVDSCommand, log id: be2058e
>2014-08-05 12:53:01,480 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [577a9eb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 12:53:01,529 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-54) [28b9964a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 12:53:01,537 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 12:53:01,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b881afd
>2014-08-05 12:53:01,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5b881afd
>2014-08-05 12:53:01,615 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Host rack storage connection was succeeded 
>2014-08-05 12:53:01,624 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 24a0d4ce
>2014-08-05 12:53:08,065 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-81) [7125e62c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 12:53:08,069 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-81) [7125e62c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 12:54:02,056 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 24a0d4ce
>2014-08-05 12:54:02,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70aa1a64
>2014-08-05 12:54:02,162 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] FINISH, SetMOMPolicyParametersVDSCommand, log id: 70aa1a64
>2014-08-05 12:54:02,171 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 12:54:02,177 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 12:54:02,196 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 12:54:08,180 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 12:54:08,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 12:54:08,187 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30de20c3
>2014-08-05 12:54:08,237 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling started: taskId = cf641d74-96d5-4d43-80a1-b258419f6a9b
>2014-08-05 12:54:09,257 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended: taskId = cf641d74-96d5-4d43-80a1-b258419f6a9b task status = finished
>2014-08-05 12:54:09,315 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended, spm status: SPM
>2014-08-05 12:54:09,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cf641d74-96d5-4d43-80a1-b258419f6a9b), log id: 20002368
>2014-08-05 12:54:09,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, HSMClearTaskVDSCommand, log id: 20002368
>2014-08-05 12:54:09,364 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68e40257, log id: 30de20c3
>2014-08-05 12:54:09,368 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 12:54:09,375 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 12:54:09,376 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-6) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 12:54:09,391 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7ac06cba
>2014-08-05 12:54:09,427 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 12:54:09,429 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d85319e
>2014-08-05 12:54:09,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d85319e
>2014-08-05 12:54:09,435 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ac06cba
>2014-08-05 12:54:09,436 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-6) Discovered no tasks on Storage Pool Default
>2014-08-05 13:18:03,125 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-05 13:18:03,130 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-05 13:18:03,138 INFO  [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7f717a80] Running command: LogoutBySessionCommand internal: false.
>2014-08-05 13:18:03,142 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [64313665] Running command: LogoutUserCommand internal: false.
>2014-08-05 13:18:03,146 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [64313665] Correlation ID: 64313665, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-05 13:23:56,157 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) [aeb0141] Principal admin::internal synchronized
>2014-08-05 13:24:25,028 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf.
>2014-08-05 14:23:56,164 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) [23c334ab] Principal admin::internal synchronized
>2014-08-05 14:24:25,046 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5497a121] Attempting to update VMs/Templates Ovf.
>2014-08-05 15:23:56,170 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized
>2014-08-05 15:24:25,080 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Attempting to update VMs/Templates Ovf.
>2014-08-05 15:58:54,530 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 15:58:54,532 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 16:01:35,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 16:01:35,476 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 16:01:35,478 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 16:01:35,480 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 16:01:35,490 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Correlation ID: 3e9bda56, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 16:03:35,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 16:03:35,531 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 16:03:35,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 16:03:35,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 16:03:35,565 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 16:03:35,583 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7dfce9c6
>2014-08-05 16:03:35,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 653a7005
>2014-08-05 16:03:35,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 16:05:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering 1 storage domains
>2014-08-05 16:05:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-05 16:05:00,006 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 16:05:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 4:05 PM
>2014-08-05 16:05:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-05 16:05:00,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a531241
>2014-08-05 16:05:35,498 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 16:05:35,501 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] FINISH, SpmStopVDSCommand, log id: 653a7005
>2014-08-05 16:05:35,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 16:05:35,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 16:05:35,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 16:05:35,513 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [29bdffa0] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 16:05:35,511 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] FINISH, ConnectStorageServerVDSCommand, log id: 1a531241
>2014-08-05 16:05:35,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] FINISH, ResetIrsVDSCommand, log id: 7dfce9c6
>2014-08-05 16:05:35,508 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [22578e65] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 16:05:35,516 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-05 16:05:35,518 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [22578e65] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 16:05:35,529 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [2978c3d] Correlation ID: 2978c3d, Job ID: 4cfc194d-cb73-41b4-aeb7-b2e81a7c24c3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 16:05:35,524 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-05 16:05:35,551 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 4:05 PM
>2014-08-05 16:05:35,557 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [3e9bda56] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 16:05:35,559 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 16:05:35,565 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [22578e65] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 16:05:35,615 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 16:05:35,634 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [231778dc] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 16:05:35,641 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [231778dc] Removing vds [] from the domain in maintenance cache
>2014-08-05 16:05:35,642 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [231778dc] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 16:05:35,643 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [3e9bda56] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 16:05:35,645 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 16:05:35,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 16:05:35,667 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 16:05:35,675 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 16:05:35,682 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 16:05:35,691 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [7204577e] Correlation ID: 7204577e, Job ID: 1424b391-6012-472c-bade-d88fc655c91a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 16:05:35,697 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [7204577e] Removing vds [] from the domain in maintenance cache
>2014-08-05 16:05:35,698 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [7204577e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 16:05:38,830 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: d5f83bd
>2014-08-05 16:05:38,889 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] FINISH, GetHardwareInfoVDSCommand, log id: d5f83bd
>2014-08-05 16:05:38,917 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [5199dc39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 16:05:38,968 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [6de7c8cb] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 16:05:38,976 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 16:05:38,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 24438db5
>2014-08-05 16:05:39,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 24438db5
>2014-08-05 16:05:39,044 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Host rack storage connection was succeeded 
>2014-08-05 16:05:39,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7208b4b0
>2014-08-05 16:05:45,517 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 16:05:45,523 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 16:06:39,497 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 7208b4b0
>2014-08-05 16:06:39,513 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3d557516
>2014-08-05 16:06:39,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3d557516
>2014-08-05 16:06:39,617 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 16:06:39,627 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 16:06:39,648 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [5937f5d4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 16:06:45,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 16:06:45,665 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 16:06:45,668 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56e233d
>2014-08-05 16:06:45,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling started: taskId = 30b57741-adb5-4d27-a2c9-29922b632e26
>2014-08-05 16:06:46,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling ended: taskId = 30b57741-adb5-4d27-a2c9-29922b632e26 task status = finished
>2014-08-05 16:06:46,785 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling ended, spm status: SPM
>2014-08-05 16:06:46,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=30b57741-adb5-4d27-a2c9-29922b632e26), log id: 26a615b9
>2014-08-05 16:06:46,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] FINISH, HSMClearTaskVDSCommand, log id: 26a615b9
>2014-08-05 16:06:46,834 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77cb28b8, log id: 56e233d
>2014-08-05 16:06:46,838 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 16:06:46,846 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 16:06:46,847 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-48) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 16:06:46,858 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4c3a9a0c
>2014-08-05 16:06:46,885 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 16:06:46,887 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 10e85e3f
>2014-08-05 16:06:46,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10e85e3f
>2014-08-05 16:06:46,893 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c3a9a0c
>2014-08-05 16:06:46,899 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] Discovered no tasks on Storage Pool Default
>2014-08-05 16:23:56,176 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) [12692311] Principal admin::internal synchronized
>2014-08-05 16:24:25,094 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [10578196] Attempting to update VMs/Templates Ovf.
>2014-08-05 17:12:20,470 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 17:23:56,182 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Principal admin::internal synchronized
>2014-08-05 17:24:25,111 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Attempting to update VMs/Templates Ovf.
>2014-08-05 18:23:56,189 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized
>2014-08-05 18:24:25,127 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Attempting to update VMs/Templates Ovf.
>2014-08-05 19:06:28,666 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 19:06:43,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 19:08:04,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 19:08:04,979 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 19:08:04,982 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-89) [5784753b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 19:08:04,984 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-89) [5784753b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 19:08:04,997 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [5784753b] Correlation ID: 5784753b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 19:10:05,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 19:10:05,029 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [5784753b] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 19:10:05,029 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 19:10:05,030 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 19:10:05,048 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 19:10:05,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3ce5e670
>2014-08-05 19:10:05,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2da35dac
>2014-08-05 19:10:05,085 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 19:12:04,999 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 19:12:05,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] FINISH, SpmStopVDSCommand, log id: 2da35dac
>2014-08-05 19:12:05,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 19:12:05,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 19:12:05,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] FINISH, ResetIrsVDSCommand, log id: 3ce5e670
>2014-08-05 19:12:05,009 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [71be4699] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 19:12:05,013 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [8557062] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 19:12:05,017 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [8557062] vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 19:12:05,021 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [8557062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 19:12:05,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] Correlation ID: 1ba0d6bc, Job ID: b615de3d-7fb9-4a12-a529-e685175fdbd1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 19:12:05,062 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [5784753b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 19:12:05,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 19:12:05,074 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [8557062] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 19:12:05,124 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 19:12:05,141 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 19:12:05,148 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] Removing vds [] from the domain in maintenance cache
>2014-08-05 19:12:05,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 19:12:05,150 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [5784753b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 19:12:05,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 19:12:05,156 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 19:12:05,171 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 19:12:05,177 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 19:12:05,182 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 19:12:05,197 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Correlation ID: 64ee7315, Job ID: 932e43e0-b76f-4b99-ad80-17ddec28570f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 19:12:05,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Removing vds [] from the domain in maintenance cache
>2014-08-05 19:12:05,202 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 19:12:08,337 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 560a8536
>2014-08-05 19:12:08,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, GetHardwareInfoVDSCommand, log id: 560a8536
>2014-08-05 19:12:08,415 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [4ffdb29c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 19:12:08,466 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [b9f6021] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 19:12:08,473 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 19:12:08,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 226de7da
>2014-08-05 19:12:08,546 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 226de7da
>2014-08-05 19:12:08,547 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] Host rack storage connection was succeeded 
>2014-08-05 19:12:08,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7319ddb5
>2014-08-05 19:12:15,019 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 19:12:15,024 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 19:13:08,958 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 7319ddb5
>2014-08-05 19:13:08,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d7c3588
>2014-08-05 19:13:09,066 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d7c3588
>2014-08-05 19:13:09,074 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2f905322] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 19:13:09,080 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2f905322] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 19:13:09,094 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [5b85dca6] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 19:13:15,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 19:13:15,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 19:13:15,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b6d70fe
>2014-08-05 19:13:15,242 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling started: taskId = 600710a5-8f0d-4c9a-8e1b-12d19a74fd62
>2014-08-05 19:13:16,249 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling ended: taskId = 600710a5-8f0d-4c9a-8e1b-12d19a74fd62 task status = finished
>2014-08-05 19:13:16,304 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling ended, spm status: SPM
>2014-08-05 19:13:16,307 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=600710a5-8f0d-4c9a-8e1b-12d19a74fd62), log id: 436b2524
>2014-08-05 19:13:16,352 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, HSMClearTaskVDSCommand, log id: 436b2524
>2014-08-05 19:13:16,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1eb1b0a7, log id: 6b6d70fe
>2014-08-05 19:13:16,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 19:13:16,364 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 19:13:16,366 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-4) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 19:13:16,377 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3239db0f
>2014-08-05 19:13:16,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 19:13:16,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 78209a91
>2014-08-05 19:13:16,413 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78209a91
>2014-08-05 19:13:16,414 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3239db0f
>2014-08-05 19:13:16,415 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-4) Discovered no tasks on Storage Pool Default
>2014-08-05 19:23:56,194 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Principal admin::internal synchronized
>2014-08-05 19:24:25,145 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) [65d9de58] Attempting to update VMs/Templates Ovf.
>2014-08-05 20:23:56,200 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) [4e09115c] Principal admin::internal synchronized
>2014-08-05 20:24:25,159 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Attempting to update VMs/Templates Ovf.
>2014-08-05 21:23:56,207 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) [686e64a1] Principal admin::internal synchronized
>2014-08-05 21:24:25,174 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf.
>2014-08-05 22:13:02,270 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-05 22:13:02,272 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-05 22:14:32,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 22:14:32,085 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 22:14:32,089 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-05 22:14:32,091 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-05 22:14:32,107 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Correlation ID: 5f3f6fb7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-05 22:16:32,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 22:16:32,133 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 22:16:32,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-05 22:16:32,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-05 22:16:32,169 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-05 22:16:32,187 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5af5ebde
>2014-08-05 22:16:32,191 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2d1531b1
>2014-08-05 22:16:32,197 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-05 22:18:32,102 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-05 22:18:32,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-99) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-05 22:18:32,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [5b85dca6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-05 22:18:32,111 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [5b85dca6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-05 22:18:32,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] FINISH, SpmStopVDSCommand, log id: 2d1531b1
>2014-08-05 22:18:32,108 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-05 22:18:32,114 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-05 22:18:32,118 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] FINISH, ResetIrsVDSCommand, log id: 5af5ebde
>2014-08-05 22:18:32,152 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-05 22:18:32,154 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] Correlation ID: 2b51aca4, Job ID: e287e501-4934-4d16-80c4-74221bf6c77a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-05 22:18:32,160 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [5f3f6fb7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 22:18:32,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-05 22:18:32,219 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-05 22:18:32,236 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [654db1ff] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-05 22:18:32,242 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [654db1ff] Removing vds [] from the domain in maintenance cache
>2014-08-05 22:18:32,243 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [654db1ff] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 22:18:32,244 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [5f3f6fb7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-05 22:18:32,246 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-05 22:18:32,250 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-05 22:18:32,264 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 22:18:32,270 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-05 22:18:32,275 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-05 22:18:32,284 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Correlation ID: 3fb49475, Job ID: 9cd5fe91-9ce7-423c-8a88-b7d8729ace8c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-05 22:18:32,289 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Removing vds [] from the domain in maintenance cache
>2014-08-05 22:18:32,290 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Removing host(s) [] from hosts unseen domain report cache
>2014-08-05 22:18:35,229 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@698df50
>2014-08-05 22:18:35,231 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] HostName = rack
>2014-08-05 22:18:35,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-05 22:18:35,247 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-05 22:18:35,254 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-05 22:18:38,484 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 520c9d01
>2014-08-05 22:18:38,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] FINISH, GetHardwareInfoVDSCommand, log id: 520c9d01
>2014-08-05 22:18:38,556 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [155dec1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 22:18:38,566 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [155dec1] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-05 22:18:38,568 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-61) [3d09986d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 22:18:38,575 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-05 22:18:38,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1b8c12a1
>2014-08-05 22:18:38,641 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1b8c12a1
>2014-08-05 22:18:38,642 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] Host rack storage connection was succeeded 
>2014-08-05 22:18:38,648 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 585e3da2
>2014-08-05 22:18:42,115 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-30) [3adcf236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 22:18:42,120 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-30) [3adcf236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-05 22:19:39,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 585e3da2
>2014-08-05 22:19:39,068 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 687d7a19
>2014-08-05 22:19:39,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 687d7a19
>2014-08-05 22:19:39,169 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [424859a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-05 22:19:39,176 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [424859a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-05 22:19:39,205 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [240369c1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-05 22:19:42,279 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-05 22:19:42,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-05 22:19:42,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56424d6d
>2014-08-05 22:19:42,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling started: taskId = fbaaa79c-16f9-4470-8dc9-f22761822adb
>2014-08-05 22:19:43,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling ended: taskId = fbaaa79c-16f9-4470-8dc9-f22761822adb task status = finished
>2014-08-05 22:19:43,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling ended, spm status: SPM
>2014-08-05 22:19:43,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fbaaa79c-16f9-4470-8dc9-f22761822adb), log id: 19951f7
>2014-08-05 22:19:43,445 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] FINISH, HSMClearTaskVDSCommand, log id: 19951f7
>2014-08-05 22:19:43,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5dc29e, log id: 56424d6d
>2014-08-05 22:19:43,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-05 22:19:43,457 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-05 22:19:43,458 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-15) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-05 22:19:43,471 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e7036e7
>2014-08-05 22:19:43,496 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-05 22:19:43,498 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6bb68551
>2014-08-05 22:19:43,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6bb68551
>2014-08-05 22:19:43,504 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e7036e7
>2014-08-05 22:19:43,505 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] Discovered no tasks on Storage Pool Default
>2014-08-05 22:23:56,213 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized
>2014-08-05 22:24:25,190 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf.
>2014-08-05 23:23:56,218 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-52) [8557062] Principal admin::internal synchronized
>2014-08-05 23:24:25,220 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf.
>2014-08-06 00:23:56,227 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-08-06 00:24:25,235 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-08-06 01:19:35,727 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 01:19:35,729 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 01:21:01,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [675f1edc] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 01:21:01,731 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 01:21:01,735 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [6e64f208] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 01:21:01,737 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [6e64f208] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 01:21:01,748 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [6e64f208] Correlation ID: 6e64f208, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 01:23:01,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 01:23:01,785 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [6e64f208] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 01:23:01,786 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 01:23:01,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 01:23:01,821 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 01:23:01,839 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 124023c5
>2014-08-06 01:23:01,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 74dafaf5
>2014-08-06 01:23:01,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 01:23:56,233 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-89) [5784753b] Principal admin::internal synchronized
>2014-08-06 01:24:25,251 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Attempting to update VMs/Templates Ovf.
>2014-08-06 01:25:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Autorecovering 1 storage domains
>2014-08-06 01:25:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-06 01:25:00,006 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 01:25:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 1:25 AM
>2014-08-06 01:25:00,017 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-06 01:25:00,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c704cb3
>2014-08-06 01:25:01,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 01:25:01,775 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 01:25:01,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 01:25:01,777 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 01:26:01,859 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-08-06 01:26:01,861 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] FINISH, SpmStopVDSCommand, log id: 74dafaf5
>2014-08-06 01:26:01,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] FINISH, ResetIrsVDSCommand, log id: 124023c5
>2014-08-06 01:26:01,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4c704cb3
>2014-08-06 01:26:01,881 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 1:26 AM
>2014-08-06 01:26:01,887 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] Correlation ID: 6fc5f0a, Job ID: a0b981be-04f8-4b11-b08e-7585a91456da, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 01:26:01,893 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6e64f208] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 01:26:01,896 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 01:26:01,959 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 01:26:01,984 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 01:26:02,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Removing vds [] from the domain in maintenance cache
>2014-08-06 01:26:02,011 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 01:26:02,012 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6e64f208] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 01:26:02,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 01:26:02,020 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 01:26:02,037 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 01:26:02,043 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 01:26:02,049 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 01:26:02,058 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [215eea0] Correlation ID: 215eea0, Job ID: c90ed536-d1b3-4e2b-b1a5-aa5980d65bcd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 01:26:02,063 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) [215eea0] Removing vds [] from the domain in maintenance cache
>2014-08-06 01:26:02,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) [215eea0] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 01:27:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 01:27:01,787 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 01:27:02,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-06 01:27:02,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Failed vds listing,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>2014-08-06 01:27:02,527 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235)
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-06 01:27:02,536 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-06 01:27:05,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2ac8372a
>2014-08-06 01:27:05,824 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] FINISH, GetHardwareInfoVDSCommand, log id: 2ac8372a
>2014-08-06 01:27:05,851 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [349ba706] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 01:27:05,900 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [349ba706] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-06 01:27:05,902 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [60eb2609] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 01:27:05,910 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 01:27:05,922 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 46cc6880
>2014-08-06 01:27:05,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 46cc6880
>2014-08-06 01:27:05,994 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] Host rack storage connection was succeeded 
>2014-08-06 01:27:06,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6124fc25
>2014-08-06 01:27:11,792 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 01:27:11,797 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 01:28:06,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 6124fc25
>2014-08-06 01:28:06,438 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d96cbbf
>2014-08-06 01:28:06,532 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1d96cbbf
>2014-08-06 01:28:06,539 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [6513a350] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 01:28:06,545 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [6513a350] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 01:28:06,558 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [6a31c23d] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 01:28:11,910 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 01:28:11,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 01:28:11,918 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2123d38a
>2014-08-06 01:28:11,965 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling started: taskId = b07bd89b-5712-4747-bd9c-2c8684c1beb3
>2014-08-06 01:28:12,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling ended: taskId = b07bd89b-5712-4747-bd9c-2c8684c1beb3 task status = finished
>2014-08-06 01:28:13,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling ended, spm status: SPM
>2014-08-06 01:28:13,029 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b07bd89b-5712-4747-bd9c-2c8684c1beb3), log id: 581b2fd9
>2014-08-06 01:28:13,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] FINISH, HSMClearTaskVDSCommand, log id: 581b2fd9
>2014-08-06 01:28:13,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6189f619, log id: 2123d38a
>2014-08-06 01:28:13,079 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 01:28:13,085 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [23c334ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 01:28:13,095 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32a00994
>2014-08-06 01:28:13,129 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 01:28:13,131 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 32b89fb4
>2014-08-06 01:28:13,136 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32b89fb4
>2014-08-06 01:28:13,137 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32a00994
>2014-08-06 01:28:13,138 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-37) [23c334ab] Discovered no tasks on Storage Pool Default
>2014-08-06 02:23:56,239 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-29) [327b509d] Principal admin::internal synchronized
>2014-08-06 02:24:25,253 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Attempting to update VMs/Templates Ovf.
>2014-08-06 03:23:56,244 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Principal admin::internal synchronized
>2014-08-06 03:24:25,267 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf.
>2014-08-06 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Start deleteAgedOutAuditLogs
>2014-08-06 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) [55405a19] Start deleteAgedOutCommandEntities
>2014-08-06 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) [55405a19] Finished deleteAgedOutCommandEntities
>2014-08-06 03:35:35,016 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Finished deleteAgedOutAuditLogs
>2014-08-06 04:23:56,249 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized
>2014-08-06 04:24:25,283 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Attempting to update VMs/Templates Ovf.
>2014-08-06 04:27:50,606 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 04:27:50,608 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 04:29:29,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 04:29:29,823 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 04:29:29,828 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 04:29:29,832 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 04:29:29,849 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Correlation ID: 39ee7ea8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 04:31:29,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 04:31:29,871 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 04:31:29,872 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 04:31:29,873 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 04:31:29,918 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 04:31:29,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2fe73741
>2014-08-06 04:31:29,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1373a335
>2014-08-06 04:31:29,944 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 04:33:29,842 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 04:33:29,845 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] FINISH, SpmStopVDSCommand, log id: 1373a335
>2014-08-06 04:33:29,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 04:33:29,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 04:33:29,848 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [12692311] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 04:33:29,851 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 04:33:29,852 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [12692311] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 04:33:29,856 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] FINISH, ResetIrsVDSCommand, log id: 2fe73741
>2014-08-06 04:33:29,885 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [12692311] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 04:33:29,891 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [24d41939] Correlation ID: 24d41939, Job ID: 831f1f82-0fdf-458a-bbfe-b00c578ead72, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 04:33:29,897 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [39ee7ea8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 04:33:29,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 04:33:29,953 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 04:33:29,969 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 04:33:29,975 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 04:33:29,980 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 04:33:29,989 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Correlation ID: 2f86b7e4, Job ID: 08f731d2-80ec-4b96-9c96-306a760a4282, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 04:33:29,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Removing vds [] from the domain in maintenance cache
>2014-08-06 04:33:29,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 04:33:29,997 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 04:33:30,001 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 04:33:30,019 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [7a434c52] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 04:33:30,025 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [7a434c52] Removing vds [] from the domain in maintenance cache
>2014-08-06 04:33:30,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [7a434c52] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 04:33:30,027 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [39ee7ea8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 04:33:33,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 23d7d633
>2014-08-06 04:33:33,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] FINISH, GetHardwareInfoVDSCommand, log id: 23d7d633
>2014-08-06 04:33:33,218 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [7e4cbe3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 04:33:33,268 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [16648d26] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 04:33:33,275 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 04:33:33,281 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ded8a61
>2014-08-06 04:33:33,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7ded8a61
>2014-08-06 04:33:33,342 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] Host rack storage connection was succeeded 
>2014-08-06 04:33:33,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6afd7327
>2014-08-06 04:33:39,858 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 04:33:39,862 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 04:34:33,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 6afd7327
>2014-08-06 04:34:33,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16c582a7
>2014-08-06 04:34:33,865 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 16c582a7
>2014-08-06 04:34:33,872 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [665369f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 04:34:33,878 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [665369f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 04:34:33,894 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [6d05f75f] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 04:34:39,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 04:34:39,984 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 04:34:39,985 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f58b41a
>2014-08-06 04:34:40,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling started: taskId = 79a8065d-89ce-4d94-9aa3-0363ad72dca5
>2014-08-06 04:34:41,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended: taskId = 79a8065d-89ce-4d94-9aa3-0363ad72dca5 task status = finished
>2014-08-06 04:34:41,096 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended, spm status: SPM
>2014-08-06 04:34:41,099 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=79a8065d-89ce-4d94-9aa3-0363ad72dca5), log id: 1e23ed72
>2014-08-06 04:34:41,143 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, HSMClearTaskVDSCommand, log id: 1e23ed72
>2014-08-06 04:34:41,144 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@588896ea, log id: 1f58b41a
>2014-08-06 04:34:41,147 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 04:34:41,153 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 04:34:41,162 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3b5dce45
>2014-08-06 04:34:41,192 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 04:34:41,194 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 50d7369b
>2014-08-06 04:34:41,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50d7369b
>2014-08-06 04:34:41,201 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b5dce45
>2014-08-06 04:34:41,202 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-33) Discovered no tasks on Storage Pool Default
>2014-08-06 05:23:56,257 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) [56fe64d8] Principal admin::internal synchronized
>2014-08-06 05:24:25,297 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) [795e8140] Attempting to update VMs/Templates Ovf.
>2014-08-06 06:23:56,264 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized
>2014-08-06 06:24:25,331 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) [6199900c] Attempting to update VMs/Templates Ovf.
>2014-08-06 07:23:56,271 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Principal admin::internal synchronized
>2014-08-06 07:24:25,345 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf.
>2014-08-06 07:34:25,187 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 07:34:25,190 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 07:35:56,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 07:35:56,501 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 07:35:56,505 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [75c8e759] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 07:35:56,507 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [75c8e759] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 07:35:56,531 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [75c8e759] Correlation ID: 75c8e759, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 07:37:56,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 07:37:56,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [75c8e759] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 07:37:56,553 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 07:37:56,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 07:37:56,587 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 07:37:56,606 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1d0db938
>2014-08-06 07:37:56,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1ada5348
>2014-08-06 07:37:56,615 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 07:39:56,519 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 07:39:56,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] FINISH, SpmStopVDSCommand, log id: 1ada5348
>2014-08-06 07:39:56,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 07:39:56,525 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 07:39:56,529 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 07:39:56,529 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 07:39:56,532 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] FINISH, ResetIrsVDSCommand, log id: 1d0db938
>2014-08-06 07:39:56,531 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 07:39:56,541 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] Correlation ID: 38bb2b88, Job ID: 9a102b86-0aa0-456d-9c0d-e517af511753, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 07:39:56,561 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 07:39:56,565 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [75c8e759] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 07:39:56,566 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 07:39:56,630 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 07:39:56,654 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 07:39:56,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] Removing vds [] from the domain in maintenance cache
>2014-08-06 07:39:56,662 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 07:39:56,663 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [75c8e759] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 07:39:56,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 07:39:56,669 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 07:39:56,684 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 07:39:56,691 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 07:39:56,697 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 07:39:56,706 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Correlation ID: 349f4a8e, Job ID: 7ad6da6d-ad2f-4aa1-9542-abddabcbf1c5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 07:39:56,711 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Removing vds [] from the domain in maintenance cache
>2014-08-06 07:39:56,712 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 07:39:59,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2ed48565
>2014-08-06 07:39:59,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, GetHardwareInfoVDSCommand, log id: 2ed48565
>2014-08-06 07:39:59,903 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [455ca1b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 07:39:59,915 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [5e70114d] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 07:39:59,924 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 07:39:59,936 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e1573a6
>2014-08-06 07:39:59,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6e1573a6
>2014-08-06 07:40:00,003 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Host rack storage connection was succeeded 
>2014-08-06 07:40:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5c182324] Autorecovering 2 storage domains
>2014-08-06 07:40:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5c182324] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-06 07:40:00,008 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 07:40:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 7:40 AM
>2014-08-06 07:40:00,010 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 790cfb1
>2014-08-06 07:40:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 7:40 AM
>2014-08-06 07:40:00,017 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1a0ac637] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-08-06 07:40:00,019 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 07:40:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 7:40 AM
>2014-08-06 07:40:00,027 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 7:40 AM
>2014-08-06 07:40:06,533 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 07:40:06,537 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 07:41:00,403 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 790cfb1
>2014-08-06 07:41:00,416 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 51329ea5
>2014-08-06 07:41:00,512 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 51329ea5
>2014-08-06 07:41:00,519 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 07:41:00,525 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 07:41:00,548 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [405deed8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 07:41:06,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 07:41:06,664 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 07:41:06,666 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ed27eb0
>2014-08-06 07:41:06,714 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling started: taskId = c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd
>2014-08-06 07:41:07,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling ended: taskId = c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd task status = finished
>2014-08-06 07:41:07,777 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling ended, spm status: SPM
>2014-08-06 07:41:07,780 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd), log id: 1db71dce
>2014-08-06 07:41:07,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] FINISH, HSMClearTaskVDSCommand, log id: 1db71dce
>2014-08-06 07:41:07,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@41b78d10, log id: 4ed27eb0
>2014-08-06 07:41:07,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 07:41:07,837 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 07:41:07,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 76d15b2a
>2014-08-06 07:41:07,881 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 07:41:07,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 236f6242
>2014-08-06 07:41:07,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 236f6242
>2014-08-06 07:41:07,889 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 76d15b2a
>2014-08-06 07:41:07,890 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] Discovered no tasks on Storage Pool Default
>2014-08-06 08:23:56,280 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized
>2014-08-06 08:24:25,360 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) [9016fac] Attempting to update VMs/Templates Ovf.
>2014-08-06 09:23:56,293 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [16080eb6] Principal admin::internal synchronized
>2014-08-06 09:24:25,401 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf.
>2014-08-06 09:56:02,856 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-06 09:56:02,864 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 09:56:02,865 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 09:59:32,378 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-06 09:59:32,383 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 09:59:32,385 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:01:05,911 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-06 10:01:05,915 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 10:01:05,916 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:02:13,431 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-06 10:02:13,436 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 10:02:13,438 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:04:42,129 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false.
>2014-08-06 10:04:42,133 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 10:04:42,192 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-06 10:07:23,428 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-06 10:07:23,434 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 10:07:23,435 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:10:24,724 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-06 10:10:24,728 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 10:10:24,729 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:16,226 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:29,106 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:29,153 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:42,076 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:42,115 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:47,825 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:12:47,895 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:13:23,441 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:13:23,479 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:14:10,640 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:14:10,681 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:14:46,878 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:14:46,918 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:15:18,424 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:15:18,460 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:17:55,956 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:17:55,992 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:03,870 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:03,907 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:13,945 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:13,987 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:24,179 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:18:24,220 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:19:02,276 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:19:03,059 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:28,268 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:28,305 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:35,791 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:35,828 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:43,390 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:22:43,432 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:23:56,298 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) [795e8140] Principal admin::internal synchronized
>2014-08-06 10:24:25,420 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [5784753b] Attempting to update VMs/Templates Ovf.
>2014-08-06 10:25:36,360 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:25:36,405 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:25:41,525 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:25:41,570 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value
>2014-08-06 10:40:58,100 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 10:40:58,102 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 10:42:23,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 10:42:23,036 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 10:42:23,040 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-47) [772ebc80] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 10:42:23,042 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-47) [772ebc80] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 10:42:23,055 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [772ebc80] Correlation ID: 772ebc80, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 10:44:23,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 10:44:23,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) [772ebc80] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 10:44:23,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 10:44:23,088 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 10:44:23,124 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 10:44:23,141 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1773d724
>2014-08-06 10:44:23,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 164ca56d
>2014-08-06 10:44:23,151 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 10:45:00,009 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Autorecovering 1 storage domains
>2014-08-06 10:45:00,010 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-06 10:45:00,012 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 10:45:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 10:45 AM
>2014-08-06 10:45:00,031 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-06 10:45:00,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e1e615b
>2014-08-06 10:46:23,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 10:46:23,064 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [55405a19] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 10:46:23,065 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 10:46:23,066 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 10:47:23,064 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 10:47:23,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 10:47:23,072 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 10:47:23,066 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 10:47:23,079 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] FINISH, ConnectStorageServerVDSCommand, log id: 4e1e615b
>2014-08-06 10:47:23,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 10:47:23,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [75c8e759] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 10:47:23,070 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] FINISH, SpmStopVDSCommand, log id: 164ca56d
>2014-08-06 10:47:23,080 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-06 10:47:23,074 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy82.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 10:47:23,090 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] FINISH, ResetIrsVDSCommand, log id: 1773d724
>2014-08-06 10:47:23,088 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-06 10:47:23,111 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 10:47 AM
>2014-08-06 10:47:23,118 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 10:47:23,122 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [6607a054] Correlation ID: 6607a054, Job ID: 34dd04ba-c423-4b37-a0eb-b57e499a5407, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 10:47:23,129 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-26) [772ebc80] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 10:47:23,131 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 10:47:23,189 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 10:47:23,207 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 10:47:23,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] Removing vds [] from the domain in maintenance cache
>2014-08-06 10:47:23,215 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 10:47:23,216 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-26) [772ebc80] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 10:47:23,218 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 10:47:23,223 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 10:47:23,237 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 10:47:23,244 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 10:47:23,249 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 10:47:23,258 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Correlation ID: 582fadb7, Job ID: ad6a13d7-d0d7-458d-987b-7cd154bcc4eb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 10:47:23,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Removing vds [] from the domain in maintenance cache
>2014-08-06 10:47:23,264 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 10:47:26,368 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 1a84ce87
>2014-08-06 10:47:26,420 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] FINISH, GetHardwareInfoVDSCommand, log id: 1a84ce87
>2014-08-06 10:47:26,444 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [3b14ddd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 10:47:26,495 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [6528240] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 10:47:26,502 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 10:47:26,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ea8369c
>2014-08-06 10:47:26,569 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3ea8369c
>2014-08-06 10:47:26,570 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Host rack storage connection was succeeded 
>2014-08-06 10:47:26,578 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 16bfcf72
>2014-08-06 10:47:33,087 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 10:47:33,090 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 10:48:26,959 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 16bfcf72
>2014-08-06 10:48:26,975 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15ef9e71
>2014-08-06 10:48:27,071 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] FINISH, SetMOMPolicyParametersVDSCommand, log id: 15ef9e71
>2014-08-06 10:48:27,078 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 10:48:27,085 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 10:48:27,099 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [6b82e961] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 10:48:33,250 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 10:48:33,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 10:48:33,258 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28679213
>2014-08-06 10:48:33,306 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling started: taskId = 28b2276c-2810-4e5a-af1f-f30a1918032a
>2014-08-06 10:48:34,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling ended: taskId = 28b2276c-2810-4e5a-af1f-f30a1918032a task status = finished
>2014-08-06 10:48:34,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling ended, spm status: SPM
>2014-08-06 10:48:34,371 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=28b2276c-2810-4e5a-af1f-f30a1918032a), log id: 4c991eb6
>2014-08-06 10:48:34,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] FINISH, HSMClearTaskVDSCommand, log id: 4c991eb6
>2014-08-06 10:48:34,417 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@545f4b93, log id: 28679213
>2014-08-06 10:48:34,421 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 10:48:34,428 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 10:48:34,440 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: e685a49
>2014-08-06 10:48:34,470 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 10:48:34,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5970fa9d
>2014-08-06 10:48:34,479 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5970fa9d
>2014-08-06 10:48:34,481 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e685a49
>2014-08-06 10:48:34,482 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] Discovered no tasks on Storage Pool Default
>2014-08-06 11:23:56,304 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Principal admin::internal synchronized
>2014-08-06 11:24:25,441 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf.
>2014-08-06 11:24:26,746 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-26) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-06 12:23:56,310 INFO  [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) [37510499] Principal admin::internal synchronized
>2014-08-06 12:24:25,457 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
>2014-08-06 12:30:52,970 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-06 12:30:52,976 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-06 12:30:52,977 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-06 12:30:52,977 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-06 12:30:52,980 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-06 12:30:52,980 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-06 12:30:52,981 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-06 12:30:52,982 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-06 12:30:52,982 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-06 12:30:52,983 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-06 12:30:52,983 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-06 12:30:52,990 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-06 12:30:52,990 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-06 12:30:52,991 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-06 12:30:52,991 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-06 12:30:52,992 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-06 12:30:52,992 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-06 12:30:52,993 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-06 12:30:52,993 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-06 12:30:52,994 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-06 12:30:52,994 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-06 12:30:52,995 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-06 12:30:53,002 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-06 12:30:53,002 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-06 12:30:53,003 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-06 12:30:53,003 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-06 12:30:53,004 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-06 12:30:53,004 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-06 12:30:53,005 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-06 12:30:53,005 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-06 12:30:53,006 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-06 12:30:53,006 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-06 12:30:53,007 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-06 12:30:53,007 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-06 12:30:53,014 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-06 12:30:53,014 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-06 12:30:53,015 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-06 12:30:53,015 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-06 12:30:53,016 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-06 12:30:53,016 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-06 12:30:53,017 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-06 12:30:53,017 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-06 12:30:53,018 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-06 12:30:53,018 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-06 12:30:53,019 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-06 12:30:53,019 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-06 12:30:53,022 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-06 12:30:53,023 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-06 12:30:53,023 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-06 12:30:53,024 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-06 12:30:53,029 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-06 12:30:53,030 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-06 12:30:53,030 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-06 12:30:53,031 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-06 12:30:53,032 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-06 12:30:53,032 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-06 12:30:53,033 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-06 12:30:53,033 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-06 12:30:53,033 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-06 12:30:53,034 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-06 12:30:53,035 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-06 12:30:53,036 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt".
>2014-08-06 12:30:53,036 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-06 12:30:53,037 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-06 12:30:53,040 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-06 12:30:53,041 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-06 12:30:53,042 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-06 12:30:53,043 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-06 12:30:53,737 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Start initializing Backend
>2014-08-06 12:30:54,203 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-4) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-06 12:30:54,311 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-06 12:30:54,311 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Start initializing dictionaries
>2014-08-06 12:30:54,314 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Finished initializing dictionaries
>2014-08-06 12:30:54,315 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing AuditLogCleanupManager
>2014-08-06 12:30:54,321 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-06 12:30:54,355 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing AuditLogCleanupManager
>2014-08-06 12:30:54,356 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing CommandEntityCleanupManager
>2014-08-06 12:30:54,358 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-06 12:30:54,363 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing CommandEntityCleanupManager
>2014-08-06 12:30:54,365 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Start initializing TagsDirector
>2014-08-06 12:30:54,374 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Tag root added to tree
>2014-08-06 12:30:54,385 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Finished initializing TagsDirector
>2014-08-06 12:30:54,386 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Start initializing IsoDomainListSyncronizer
>2014-08-06 12:30:54,393 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Finished initializing IsoDomainListSyncronizer
>2014-08-06 12:30:54,517 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Completed initializing handlers
>2014-08-06 12:30:54,544 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
>2014-08-06 12:30:54,563 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-06 12:30:54,564 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
>2014-08-06 12:30:54,565 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
>2014-08-06 12:30:54,565 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
>2014-08-06 12:30:54,566 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Mark incomplete jobs as UNKNOWN
>2014-08-06 12:30:54,594 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Start initializing JobRepositoryCleanupManager
>2014-08-06 12:30:54,595 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Finished initializing JobRepositoryCleanupManager
>2014-08-06 12:30:54,595 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Start initializing AutoRecoveryManager
>2014-08-06 12:30:54,597 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Finished initializing AutoRecoveryManager
>2014-08-06 12:30:54,599 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Start initializing ExecutionMessageDirector
>2014-08-06 12:30:54,601 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Finished initializing ExecutionMessageDirector
>2014-08-06 12:30:54,656 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Start of enabled extensions list
>2014-08-06 12:30:54,657 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-06 12:30:54,658 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-06 12:30:54,659 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) End of enabled extensions list
>2014-08-06 12:30:54,747 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager
>2014-08-06 12:30:54,756 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager
>2014-08-06 12:30:54,780 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully.
>2014-08-06 12:30:54,785 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager
>2014-08-06 12:30:54,805 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-06 12:30:54,877 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor
>2014-08-06 12:30:54,988 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-06 12:30:55,205 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-06 12:30:55,254 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager
>2014-08-06 12:30:55,264 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully.
>2014-08-06 12:30:55,265 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer
>2014-08-06 12:30:55,266 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer
>2014-08-06 12:30:55,267 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check
>2014-08-06 12:30:55,268 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check
>2014-08-06 12:30:55,283 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-08-06 12:30:55,291 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities
>2014-08-06 12:30:55,292 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities
>2014-08-06 12:30:55,307 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager
>2014-08-06 12:30:55,320 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 253
>2014-08-06 12:30:55,376 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped
>2014-08-06 12:30:55,377 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager
>2014-08-06 12:30:55,378 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat
>2014-08-06 12:30:55,385 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized
>2014-08-06 12:30:58,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ef8cbb
>2014-08-06 12:30:58,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: ef8cbb
>2014-08-06 12:30:59,284 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-06 12:30:59,450 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [17385311] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 12:31:05,354 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 12:31:05,379 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 12:31:05,395 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 12:31:05,419 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-06 12:31:05,432 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: UNASSIGNED not exist in string table
>2014-08-06 12:31:05,433 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-06 12:31:05,434 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-06 12:31:05,435 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-06 12:31:05,436 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-06 12:31:05,437 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-06 12:31:05,438 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-06 12:31:05,440 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-06 12:31:05,441 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-06 12:31:05,442 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-06 12:31:05,446 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_STOPPED not exist in string table
>2014-08-06 12:31:05,447 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_STARTED not exist in string table
>2014-08-06 12:31:05,448 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_ERROR not exist in string table
>2014-08-06 12:31:05,450 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-06 12:31:05,451 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-06 12:31:05,458 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-06 12:31:05,459 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-06 12:31:05,798 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Correlation ID: 6c6ade78, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-06 12:31:05,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [6c6ade78] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-06 12:31:05,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 12:31:05,955 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 12:31:05,978 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 161f87e5
>2014-08-06 12:31:06,027 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 12:31:06,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 72f183c3
>2014-08-06 12:31:06,042 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72f183c3
>2014-08-06 12:31:06,043 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 161f87e5
>2014-08-06 12:31:06,044 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] Discovered no tasks on Storage Pool Default
>2014-08-06 12:31:48,319 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-06 12:31:48,329 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-06 13:14:56,938 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-1) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-08-06 13:15:12,074 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-6) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-08-06 13:30:54,829 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized
>2014-08-06 13:30:55,264 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf.
>2014-08-06 13:48:14,475 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 13:48:29,604 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 13:49:54,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 13:49:54,074 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 13:49:54,085 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 13:49:54,087 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 13:49:54,103 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Correlation ID: ba1ce2a, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 13:51:54,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 13:51:54,113 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 13:51:54,115 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 13:51:54,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 13:51:54,234 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 13:51:54,307 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1e2cce75
>2014-08-06 13:51:54,313 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5a24a037
>2014-08-06 13:51:54,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 13:53:54,080 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 13:53:54,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] FINISH, SpmStopVDSCommand, log id: 5a24a037
>2014-08-06 13:53:54,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] FINISH, ResetIrsVDSCommand, log id: 1e2cce75
>2014-08-06 13:53:54,093 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-74) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 13:53:54,101 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 13:53:54,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 13:53:54,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 13:53:54,103 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 13:53:54,148 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 13:53:54,149 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [663debf6] Correlation ID: 663debf6, Job ID: 189806e5-27f6-490f-9007-79d6f6e94e18, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 13:53:54,173 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [ba1ce2a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 13:53:54,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 13:53:54,245 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 13:53:54,272 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 13:53:54,284 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] Removing vds [] from the domain in maintenance cache
>2014-08-06 13:53:54,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 13:53:54,286 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [ba1ce2a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 13:53:54,288 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 13:53:54,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 13:53:54,313 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 13:53:54,323 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 13:53:54,350 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 13:53:54,369 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Correlation ID: 7c8b8632, Job ID: 898f2d4c-9716-4c8b-bbe0-9d7f5acc25bf, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 13:53:54,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Removing vds [] from the domain in maintenance cache
>2014-08-06 13:53:54,376 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 13:53:57,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 14120542
>2014-08-06 13:53:57,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GetHardwareInfoVDSCommand, log id: 14120542
>2014-08-06 13:53:57,592 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [34e4d0fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 13:53:57,640 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [1908f1df] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 13:53:57,668 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 13:53:57,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12e153fe
>2014-08-06 13:53:57,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 12e153fe
>2014-08-06 13:53:57,727 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] Host rack storage connection was succeeded 
>2014-08-06 13:53:57,741 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4ef7b96b
>2014-08-06 13:54:04,119 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-11) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 13:54:04,124 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-11) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 13:54:58,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, ConnectStoragePoolVDSCommand, log id: 4ef7b96b
>2014-08-06 13:54:58,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7eb8dd85
>2014-08-06 13:54:58,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7eb8dd85
>2014-08-06 13:54:58,261 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [61b8688e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 13:54:58,270 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [61b8688e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 13:54:58,301 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [6bdb6450] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 13:55:04,269 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 13:55:04,282 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 13:55:04,285 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f376c77
>2014-08-06 13:55:04,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling started: taskId = 9047e6c8-834e-4666-80c1-6158df8b31df
>2014-08-06 13:55:05,342 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling ended: taskId = 9047e6c8-834e-4666-80c1-6158df8b31df task status = finished
>2014-08-06 13:55:05,398 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling ended, spm status: SPM
>2014-08-06 13:55:05,400 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=9047e6c8-834e-4666-80c1-6158df8b31df), log id: 5b99dc3b
>2014-08-06 13:55:05,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, HSMClearTaskVDSCommand, log id: 5b99dc3b
>2014-08-06 13:55:05,447 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@472edc0c, log id: 4f376c77
>2014-08-06 13:55:05,451 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 13:55:05,457 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 13:55:05,467 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2b89ae74
>2014-08-06 13:55:05,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 13:55:05,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 27789e79
>2014-08-06 13:55:05,508 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27789e79
>2014-08-06 13:55:05,509 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b89ae74
>2014-08-06 13:55:05,511 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) Discovered no tasks on Storage Pool Default
>2014-08-06 14:30:54,837 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-06 14:30:55,356 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf.
>2014-08-06 15:30:54,844 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized
>2014-08-06 15:30:55,374 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-08-06 16:30:54,851 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-08-06 16:30:55,403 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf.
>2014-08-06 16:54:53,193 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 16:54:53,197 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 16:56:25,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 16:56:25,535 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 16:56:25,540 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [6171df94] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 16:56:25,542 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [6171df94] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 16:56:25,557 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [6171df94] Correlation ID: 6171df94, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 16:58:25,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 16:58:25,585 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [6171df94] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 16:58:25,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 16:58:25,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 16:58:25,625 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 16:58:25,649 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 61ffa4ff
>2014-08-06 16:58:25,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2aea7969
>2014-08-06 16:58:25,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 17:00:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering 1 storage domains
>2014-08-06 17:00:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-06 17:00:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 17:00:00,020 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 5:00 PM
>2014-08-06 17:00:00,100 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-06 17:00:00,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 287108d7
>2014-08-06 17:00:25,557 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 17:00:25,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 17:00:25,559 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 17:00:25,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 17:01:25,560 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 17:01:25,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] FINISH, SpmStopVDSCommand, log id: 2aea7969
>2014-08-06 17:01:25,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 17:01:25,566 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 17:01:25,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 17:01:25,579 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 17:01:25,577 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] FINISH, ConnectStorageServerVDSCommand, log id: 287108d7
>2014-08-06 17:01:25,583 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-06 17:01:25,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] FINISH, ResetIrsVDSCommand, log id: 61ffa4ff
>2014-08-06 17:01:25,596 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 17:01:25,597 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) vds::refreshVdsStats Failed getVdsStats,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 17:01:25,605 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [59cec128] Correlation ID: 59cec128, Job ID: 29d836ca-f288-40b0-a86d-3569b69bacf1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 17:01:25,609 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-06 17:01:25,611 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 5:01 PM
>2014-08-06 17:01:25,605 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.getVdsStats(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 17:01:25,616 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6171df94] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 17:01:25,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 17:01:25,645 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 17:01:25,704 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 17:01:25,724 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [401e1c49] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 17:01:25,733 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [401e1c49] Removing vds [] from the domain in maintenance cache
>2014-08-06 17:01:25,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [401e1c49] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 17:01:25,735 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6171df94] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 17:01:25,737 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 17:01:25,743 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 17:01:25,760 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 17:01:25,769 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 17:01:25,776 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 17:01:25,787 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Correlation ID: 3378f4bb, Job ID: 250ea4f2-24ce-4b5e-8fdb-00f7474e11b4, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 17:01:25,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Removing vds [] from the domain in maintenance cache
>2014-08-06 17:01:25,793 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 17:01:28,927 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3d95fa3
>2014-08-06 17:01:28,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 3d95fa3
>2014-08-06 17:01:29,007 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [557965d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 17:01:29,060 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [1b1209d3] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 17:01:29,070 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 17:01:29,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b31f43f
>2014-08-06 17:01:29,145 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7b31f43f
>2014-08-06 17:01:29,146 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Host rack storage connection was succeeded 
>2014-08-06 17:01:29,154 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 47a5a658
>2014-08-06 17:01:35,586 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 17:01:35,589 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 17:02:29,573 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 47a5a658
>2014-08-06 17:02:29,585 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 18c000f0
>2014-08-06 17:02:29,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 18c000f0
>2014-08-06 17:02:29,686 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 17:02:29,694 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 17:02:29,712 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [587e61f3] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 17:02:35,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 17:02:35,719 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 17:02:35,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 778bc1f5
>2014-08-06 17:02:35,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling started: taskId = f716ddd6-20e5-44f2-8bb3-fcf92819a37e
>2014-08-06 17:02:36,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling ended: taskId = f716ddd6-20e5-44f2-8bb3-fcf92819a37e task status = finished
>2014-08-06 17:02:36,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling ended, spm status: SPM
>2014-08-06 17:02:36,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-56) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f716ddd6-20e5-44f2-8bb3-fcf92819a37e), log id: 5c002b4
>2014-08-06 17:02:36,892 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, HSMClearTaskVDSCommand, log id: 5c002b4
>2014-08-06 17:02:36,893 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f44897, log id: 778bc1f5
>2014-08-06 17:02:36,899 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 17:02:36,905 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 17:02:36,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 50716646
>2014-08-06 17:02:36,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 17:02:36,947 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 146488c4
>2014-08-06 17:02:36,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 146488c4
>2014-08-06 17:02:36,956 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50716646
>2014-08-06 17:02:36,957 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-9) Discovered no tasks on Storage Pool Default
>2014-08-06 17:30:54,859 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized
>2014-08-06 17:30:55,421 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf.
>2014-08-06 18:30:54,867 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) [724f655c] Principal admin::internal synchronized
>2014-08-06 18:30:55,457 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf.
>2014-08-06 19:30:54,874 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized
>2014-08-06 19:30:55,470 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
>2014-08-06 20:02:26,159 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 20:02:26,162 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 20:03:53,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 20:03:53,284 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 20:03:53,287 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 20:03:53,289 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 20:03:53,306 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Correlation ID: 6bd72a89, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 20:05:53,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 20:05:53,334 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [6bd72a89] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 20:05:53,335 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 20:05:53,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 20:05:53,372 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 20:05:53,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: fa726cd
>2014-08-06 20:05:53,394 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 59322024
>2014-08-06 20:05:53,401 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 20:07:53,306 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 20:07:53,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] FINISH, SpmStopVDSCommand, log id: 59322024
>2014-08-06 20:07:53,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-73) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 20:07:53,312 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 20:07:53,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 20:07:53,315 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] FINISH, ResetIrsVDSCommand, log id: fa726cd
>2014-08-06 20:07:53,316 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 20:07:53,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 20:07:53,333 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] Correlation ID: 1f9ab2c5, Job ID: 2149a573-d6d9-453e-889b-bc1e6b21ae08, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 20:07:53,347 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 20:07:53,349 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [6bd72a89] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 20:07:53,351 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 20:07:53,417 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 20:07:53,436 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 20:07:53,443 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] Removing vds [] from the domain in maintenance cache
>2014-08-06 20:07:53,444 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 20:07:53,445 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [6bd72a89] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 20:07:53,447 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 20:07:53,452 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 20:07:53,469 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 20:07:53,477 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 20:07:53,483 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 20:07:53,495 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Correlation ID: 9899ff9, Job ID: 86c0d1b2-8b9d-4351-b998-e267423d2e7a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 20:07:53,501 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Removing vds [] from the domain in maintenance cache
>2014-08-06 20:07:53,502 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 20:07:56,605 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 53980f7b
>2014-08-06 20:07:56,656 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GetHardwareInfoVDSCommand, log id: 53980f7b
>2014-08-06 20:07:56,689 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [23e2752a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 20:07:56,702 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-100) [376829de] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 20:07:56,716 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 20:07:56,730 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c7563f7
>2014-08-06 20:07:56,792 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6c7563f7
>2014-08-06 20:07:56,793 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] Host rack storage connection was succeeded 
>2014-08-06 20:07:56,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4570b3b8
>2014-08-06 20:08:03,345 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 20:08:03,348 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 20:08:57,213 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 4570b3b8
>2014-08-06 20:08:57,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a0f0404
>2014-08-06 20:08:57,318 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a0f0404
>2014-08-06 20:08:57,326 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [2958405c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 20:08:57,332 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [2958405c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 20:08:57,349 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 20:09:03,468 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 20:09:03,473 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 20:09:03,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 57f5b162
>2014-08-06 20:09:03,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling started: taskId = 29c76088-e3cc-40cf-b745-f462e7f5c7bd
>2014-08-06 20:09:04,530 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended: taskId = 29c76088-e3cc-40cf-b745-f462e7f5c7bd task status = finished
>2014-08-06 20:09:04,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended, spm status: SPM
>2014-08-06 20:09:04,591 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=29c76088-e3cc-40cf-b745-f462e7f5c7bd), log id: 648e7d39
>2014-08-06 20:09:04,636 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 648e7d39
>2014-08-06 20:09:04,637 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4eab0323, log id: 57f5b162
>2014-08-06 20:09:04,641 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 20:09:04,648 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 20:09:04,651 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue.
>2014-08-06 20:09:04,661 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5b02294
>2014-08-06 20:09:04,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 20:09:04,692 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6620e161
>2014-08-06 20:09:04,698 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6620e161
>2014-08-06 20:09:04,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b02294
>2014-08-06 20:09:04,699 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) Discovered no tasks on Storage Pool Default
>2014-08-06 20:30:54,881 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized
>2014-08-06 20:30:55,492 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
>2014-08-06 21:29:22,049 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-06 21:30:54,887 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-38) Principal admin::internal synchronized
>2014-08-06 21:30:55,520 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
>2014-08-06 22:30:54,893 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized
>2014-08-06 22:30:55,537 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf.
>2014-08-06 23:08:46,527 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-06 23:08:46,529 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-06 23:10:19,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 23:10:19,596 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 23:10:19,600 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [699d5b35] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-06 23:10:19,602 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [699d5b35] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-06 23:10:19,621 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [699d5b35] Correlation ID: 699d5b35, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-06 23:12:19,638 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 23:12:19,640 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [699d5b35] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 23:12:19,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-06 23:12:19,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-06 23:12:19,701 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-06 23:12:19,728 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7bd8f602
>2014-08-06 23:12:19,732 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1957f05e
>2014-08-06 23:12:19,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-06 23:14:19,617 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-06 23:14:19,620 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] FINISH, SpmStopVDSCommand, log id: 1957f05e
>2014-08-06 23:14:19,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-06 23:14:19,623 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-06 23:14:19,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-43) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-06 23:14:19,628 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-06 23:14:19,629 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-06 23:14:19,627 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] FINISH, ResetIrsVDSCommand, log id: 7bd8f602
>2014-08-06 23:14:19,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [12f9d30] Correlation ID: 12f9d30, Job ID: 023e5721-b319-42d0-95fc-9e7016679777, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-06 23:14:19,668 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-06 23:14:19,674 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [699d5b35] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 23:14:19,676 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-06 23:14:19,739 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-06 23:14:19,760 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-06 23:14:19,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] Removing vds [] from the domain in maintenance cache
>2014-08-06 23:14:19,769 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 23:14:19,769 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [699d5b35] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-06 23:14:19,771 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-06 23:14:19,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-06 23:14:19,792 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 23:14:19,798 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-06 23:14:19,805 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-06 23:14:19,827 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Correlation ID: 3893d5cd, Job ID: 1f799972-e1d4-4606-b535-3f01510c18ff, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-06 23:14:19,832 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Removing vds [] from the domain in maintenance cache
>2014-08-06 23:14:19,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Removing host(s) [] from hosts unseen domain report cache
>2014-08-06 23:14:22,928 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 230a6365
>2014-08-06 23:14:22,977 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 230a6365
>2014-08-06 23:14:23,011 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [3dc0f67d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 23:14:23,025 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [10eae19] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 23:14:23,037 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-06 23:14:23,047 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4973249
>2014-08-06 23:14:23,109 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4973249
>2014-08-06 23:14:23,110 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] Host rack storage connection was succeeded 
>2014-08-06 23:14:23,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 23a021b3
>2014-08-06 23:14:29,631 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 23:14:29,634 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-06 23:15:23,524 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 23a021b3
>2014-08-06 23:15:23,538 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 303fd306
>2014-08-06 23:15:23,633 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 303fd306
>2014-08-06 23:15:23,643 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [327de80e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-06 23:15:23,651 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [327de80e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-06 23:15:23,677 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [57c3242a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-06 23:15:29,786 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-06 23:15:29,791 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-06 23:15:29,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 167050c4
>2014-08-06 23:15:29,842 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling started: taskId = f088c5f5-7c5a-409b-b852-c2f3a3f27f4f
>2014-08-06 23:15:30,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling ended: taskId = f088c5f5-7c5a-409b-b852-c2f3a3f27f4f task status = finished
>2014-08-06 23:15:30,906 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling ended, spm status: SPM
>2014-08-06 23:15:30,908 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f088c5f5-7c5a-409b-b852-c2f3a3f27f4f), log id: f833712
>2014-08-06 23:15:30,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] FINISH, HSMClearTaskVDSCommand, log id: f833712
>2014-08-06 23:15:30,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3cf346ed, log id: 167050c4
>2014-08-06 23:15:30,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-06 23:15:30,965 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-06 23:15:30,966 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-06 23:15:30,981 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 376e2e1e
>2014-08-06 23:15:31,015 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-06 23:15:31,016 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23a630b6
>2014-08-06 23:15:31,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23a630b6
>2014-08-06 23:15:31,022 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 376e2e1e
>2014-08-06 23:15:31,023 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] Discovered no tasks on Storage Pool Default
>2014-08-06 23:30:54,903 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized
>2014-08-06 23:30:55,550 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf.
>2014-08-07 00:30:54,915 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized
>2014-08-07 00:30:55,583 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-08-07 01:30:54,921 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized
>2014-08-07 01:30:55,612 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf.
>2014-08-07 02:00:41,521 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-07 02:15:21,896 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-07 02:15:21,900 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-07 02:16:47,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6bdb6450] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 02:16:47,584 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 02:16:47,590 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 02:16:47,592 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 02:16:47,633 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Correlation ID: 2bb2f45d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-07 02:18:47,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 02:18:47,637 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 02:18:47,638 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 02:18:47,639 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 02:18:47,680 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 02:18:47,707 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7c3c3233
>2014-08-07 02:18:47,711 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 7944a00a
>2014-08-07 02:18:47,716 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-07 02:20:00,005 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) Autorecovering 1 storage domains
>2014-08-07 02:20:00,007 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-07 02:20:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 02:20:00,011 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:20 AM
>2014-08-07 02:20:00,022 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 02:20:00,027 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5061cfd3
>2014-08-07 02:20:47,606 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-07 02:20:47,608 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] FINISH, SpmStopVDSCommand, log id: 7944a00a
>2014-08-07 02:20:47,613 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-07 02:20:47,618 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 02:20:47,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 02:20:47,620 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-07 02:20:47,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-07 02:20:47,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 02:20:47,626 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] FINISH, ResetIrsVDSCommand, log id: 7c3c3233
>2014-08-07 02:20:47,647 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] FINISH, ConnectStorageServerVDSCommand, log id: 5061cfd3
>2014-08-07 02:20:47,649 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-07 02:20:47,674 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [64444835] Correlation ID: 64444835, Job ID: 4eea4f15-b99f-4535-94bc-03e53ed61c70, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-07 02:20:47,679 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [2bb2f45d] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 02:20:47,681 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-07 02:20:47,691 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-07 02:20:47,692 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:20 AM
>2014-08-07 02:20:47,696 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-07 02:20:47,767 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-07 02:20:47,786 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1b36501d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-07 02:20:47,794 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [1b36501d] Removing vds [] from the domain in maintenance cache
>2014-08-07 02:20:47,795 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [1b36501d] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 02:20:47,796 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [2bb2f45d] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 02:20:47,798 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-07 02:20:47,804 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-07 02:20:47,821 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 02:20:47,828 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 02:20:47,833 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 02:20:47,843 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Correlation ID: 6c88dc22, Job ID: 6cd6d2cb-3320-4e6a-a07a-be504492be85, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-07 02:20:47,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Removing vds [] from the domain in maintenance cache
>2014-08-07 02:20:47,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 02:20:50,964 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: d6d2d1c
>2014-08-07 02:20:51,021 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GetHardwareInfoVDSCommand, log id: d6d2d1c
>2014-08-07 02:20:51,052 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [17cc77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 02:20:51,104 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [1d91d6a5] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 02:20:51,113 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 02:20:51,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 34583e2
>2014-08-07 02:20:51,183 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 34583e2
>2014-08-07 02:20:51,184 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] Host rack storage connection was succeeded 
>2014-08-07 02:20:51,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1b52c375
>2014-08-07 02:20:57,623 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 02:20:57,627 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 02:21:51,588 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 1b52c375
>2014-08-07 02:21:51,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3eab5140
>2014-08-07 02:21:51,700 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3eab5140
>2014-08-07 02:21:51,709 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [515c423b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 02:21:51,715 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [515c423b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 02:21:51,734 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [5b995d2] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 02:21:57,740 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 02:21:57,745 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 02:21:57,747 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c540c37
>2014-08-07 02:21:57,818 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling started: taskId = 6571e5b5-148c-4dc4-849e-f0a184206957
>2014-08-07 02:21:58,825 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling ended: taskId = 6571e5b5-148c-4dc4-849e-f0a184206957 task status = finished
>2014-08-07 02:21:58,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling ended, spm status: SPM
>2014-08-07 02:21:58,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-18) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6571e5b5-148c-4dc4-849e-f0a184206957), log id: 5aa8a24c
>2014-08-07 02:21:58,930 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, HSMClearTaskVDSCommand, log id: 5aa8a24c
>2014-08-07 02:21:58,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6896d515, log id: 5c540c37
>2014-08-07 02:21:58,936 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 02:21:58,945 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 02:21:58,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: bd502cd
>2014-08-07 02:21:58,990 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 02:21:58,993 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1631d787
>2014-08-07 02:21:58,998 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1631d787
>2014-08-07 02:21:58,999 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: bd502cd
>2014-08-07 02:21:59,000 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default
>2014-08-07 02:30:54,926 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized
>2014-08-07 02:30:55,626 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
>2014-08-07 03:30:54,937 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized
>2014-08-07 03:30:55,643 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-08-07 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-93) Start deleteAgedOutAuditLogs
>2014-08-07 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutCommandEntities
>2014-08-07 03:35:35,033 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-93) Finished deleteAgedOutAuditLogs
>2014-08-07 03:35:35,038 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutCommandEntities
>2014-08-07 04:30:54,944 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized
>2014-08-07 04:30:55,676 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf.
>2014-08-07 05:21:41,345 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-07 05:21:41,348 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-07 05:23:14,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:23:14,264 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 05:23:14,274 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 05:23:14,277 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 05:23:14,292 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Correlation ID: 6d1d4af0, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-07 05:25:14,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:25:14,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 05:25:14,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 05:25:14,308 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:25:14,354 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 05:25:14,375 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 59b92999
>2014-08-07 05:25:14,379 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 22a7ce4f
>2014-08-07 05:25:14,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-07 05:27:14,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:27:14,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 05:27:14,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 05:27:14,283 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:28:14,387 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException
>2014-08-07 05:28:14,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] FINISH, SpmStopVDSCommand, log id: 22a7ce4f
>2014-08-07 05:28:14,396 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] FINISH, ResetIrsVDSCommand, log id: 59b92999
>2014-08-07 05:28:14,423 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [41ce916] Correlation ID: 41ce916, Job ID: a0dc1eea-9aba-4ce2-8479-af836043beee, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-07 05:28:14,431 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [6d1d4af0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 05:28:14,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-07 05:28:14,531 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-07 05:28:14,564 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [3143c641] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 05:28:14,598 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [3143c641] Removing vds [] from the domain in maintenance cache
>2014-08-07 05:28:14,599 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [3143c641] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 05:28:14,600 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [6d1d4af0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 05:28:14,608 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-07 05:28:14,614 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-07 05:28:14,631 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 05:28:14,638 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 05:28:14,644 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 05:28:14,654 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Correlation ID: 2ab57563, Job ID: 868327ef-28fb-4a80-8fbc-6dc60a52f9f9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-07 05:28:14,659 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Removing vds [] from the domain in maintenance cache
>2014-08-07 05:28:14,660 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 05:29:14,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:29:14,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 05:29:14,292 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 05:29:14,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 05:29:14,329 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-3) [30318db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 05:29:14,357 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-4) Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 05:30:00,009 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Autorecovering 2 storage domains
>2014-08-07 05:30:00,011 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-08-07 05:30:00,014 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 05:30:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 5:30 AM
>2014-08-07 05:30:00,029 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 05:30:00,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 758a4431
>2014-08-07 05:30:54,952 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized
>2014-08-07 05:30:55,691 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-08-07 05:31:00,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 758a4431
>2014-08-07 05:31:00,054 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 5:31 AM
>2014-08-07 05:31:00,056 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [14b3cca1] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-07 05:31:00,058 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 05:31:00,059 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 5:31 AM
>2014-08-07 05:31:00,068 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 05:31:00,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64e390b6
>2014-08-07 05:31:14,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-69) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 05:31:14,307 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rack is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued.
>2014-08-07 05:31:14,308 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-07 05:31:14,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-07 05:31:14,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-07 05:31:14,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 05:31:14,353 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] FINISH, ConnectStorageServerVDSCommand, log id: 64e390b6
>2014-08-07 05:31:14,356 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-07 05:31:14,356 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022)
>2014-08-07 05:31:14,363 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-07 05:31:14,364 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 5:31 AM
>2014-08-07 05:31:17,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: de53fef
>2014-08-07 05:31:17,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, GetHardwareInfoVDSCommand, log id: de53fef
>2014-08-07 05:31:17,724 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [375a3aa3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 05:31:17,740 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-94) [188e29b7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 05:31:17,749 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 05:31:17,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7d6272c3
>2014-08-07 05:31:17,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7d6272c3
>2014-08-07 05:31:17,822 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] Host rack storage connection was succeeded 
>2014-08-07 05:31:17,831 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 43449517
>2014-08-07 05:31:24,359 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 05:31:24,362 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 05:32:18,227 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 43449517
>2014-08-07 05:32:18,241 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4161db52
>2014-08-07 05:32:18,333 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4161db52
>2014-08-07 05:32:18,344 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [6e904378] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 05:32:18,352 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [6e904378] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 05:32:18,381 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [a61d6df] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 05:32:24,503 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 05:32:24,508 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 05:32:24,510 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d84dfa5
>2014-08-07 05:32:24,559 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling started: taskId = 95bb7504-b263-4a49-8fd5-5810b3073302
>2014-08-07 05:32:25,566 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling ended: taskId = 95bb7504-b263-4a49-8fd5-5810b3073302 task status = finished
>2014-08-07 05:32:25,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling ended, spm status: SPM
>2014-08-07 05:32:25,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=95bb7504-b263-4a49-8fd5-5810b3073302), log id: 315052a
>2014-08-07 05:32:25,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, HSMClearTaskVDSCommand, log id: 315052a
>2014-08-07 05:32:25,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29ccb33f, log id: d84dfa5
>2014-08-07 05:32:25,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 05:32:25,686 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 05:32:25,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2ac73b48
>2014-08-07 05:32:25,732 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 05:32:25,735 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7e12fcdb
>2014-08-07 05:32:25,742 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e12fcdb
>2014-08-07 05:32:25,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2ac73b48
>2014-08-07 05:32:25,745 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) Discovered no tasks on Storage Pool Default
>2014-08-07 06:30:54,958 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized
>2014-08-07 06:30:55,693 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf.
>2014-08-07 07:30:54,968 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-30) Principal admin::internal synchronized
>2014-08-07 07:30:55,709 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-07 08:30:54,976 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized
>2014-08-07 08:30:55,743 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-08-07 08:32:01,433 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-07 08:32:16,555 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-07 08:33:39,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 08:33:39,059 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 08:33:39,062 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [57c471e1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 08:33:39,065 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [57c471e1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 08:33:39,083 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [57c471e1] Correlation ID: 57c471e1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-07 08:35:39,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 08:35:39,105 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [57c471e1] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 08:35:39,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 08:35:39,108 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 08:35:39,153 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 08:35:39,176 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: d035289
>2014-08-07 08:35:39,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6c90db65
>2014-08-07 08:35:39,225 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-07 08:37:39,075 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-07 08:37:39,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] FINISH, SpmStopVDSCommand, log id: 6c90db65
>2014-08-07 08:37:39,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-07 08:37:39,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] FINISH, ResetIrsVDSCommand, log id: d035289
>2014-08-07 08:37:39,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-3) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 08:37:39,099 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-07 08:37:39,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 08:37:39,101 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-07 08:37:39,121 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] Correlation ID: 5e9ac41c, Job ID: a99e3afe-91a6-425b-bdd4-d7c034ab6dc7, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-07 08:37:39,152 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [57c471e1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 08:37:39,154 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-07 08:37:39,163 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-07 08:37:39,227 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-07 08:37:39,247 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-07 08:37:39,254 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] Removing vds [] from the domain in maintenance cache
>2014-08-07 08:37:39,255 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 08:37:39,255 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [57c471e1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 08:37:39,257 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-07 08:37:39,262 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-07 08:37:39,279 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 08:37:39,290 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 08:37:39,302 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 08:37:39,316 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Correlation ID: 3915e0d8, Job ID: a49c7a54-2313-4558-9f1b-09202c86e30a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-07 08:37:39,321 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Removing vds [] from the domain in maintenance cache
>2014-08-07 08:37:39,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 08:37:42,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value 
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40b42e09
>2014-08-07 08:37:42,255 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) HostName = rack
>2014-08-07 08:37:42,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing
>2014-08-07 08:37:42,275 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96)
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56)
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628)
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485)
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236)
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing
>2014-08-07 08:37:42,282 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing.
>2014-08-07 08:37:45,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 70520d0a
>2014-08-07 08:37:45,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GetHardwareInfoVDSCommand, log id: 70520d0a
>2014-08-07 08:37:45,614 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [384faa36] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 08:37:45,624 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) [384faa36] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing
>2014-08-07 08:37:45,626 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [20c96164] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 08:37:45,641 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 08:37:45,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 55683a48
>2014-08-07 08:37:45,718 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 55683a48
>2014-08-07 08:37:45,719 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Host rack storage connection was succeeded 
>2014-08-07 08:37:45,728 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 45531e49
>2014-08-07 08:37:49,113 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 08:37:49,118 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 08:38:46,161 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 45531e49
>2014-08-07 08:38:46,175 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4f2ecd20
>2014-08-07 08:38:46,269 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4f2ecd20
>2014-08-07 08:38:46,275 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 08:38:46,283 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 08:38:46,303 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [5ff62557] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 08:38:49,238 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 08:38:49,244 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 08:38:49,246 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4424f251
>2014-08-07 08:38:49,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling started: taskId = f089e1c2-6576-4d86-9f38-6fedbb8dc7f3
>2014-08-07 08:38:50,301 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling ended: taskId = f089e1c2-6576-4d86-9f38-6fedbb8dc7f3 task status = finished
>2014-08-07 08:38:50,356 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling ended, spm status: SPM
>2014-08-07 08:38:50,359 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f089e1c2-6576-4d86-9f38-6fedbb8dc7f3), log id: 2946eb35
>2014-08-07 08:38:50,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] FINISH, HSMClearTaskVDSCommand, log id: 2946eb35
>2014-08-07 08:38:50,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a10977a, log id: 4424f251
>2014-08-07 08:38:50,410 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 08:38:50,420 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 08:38:50,432 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1e9994ae
>2014-08-07 08:38:50,463 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 08:38:50,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 405a044b
>2014-08-07 08:38:50,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 405a044b
>2014-08-07 08:38:50,474 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e9994ae
>2014-08-07 08:38:50,476 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] Discovered no tasks on Storage Pool Default
>2014-08-07 09:29:22,699 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 09:29:22,705 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 09:30:54,982 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized
>2014-08-07 09:30:55,758 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf.
>2014-08-07 09:53:13,531 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 09:53:13,541 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 09:55:18,568 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-3) [59a10b9a] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-08-07 09:55:18,779 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Running command: RunVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-08-07 09:55:18,786 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] START, ResumeVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 626248c8
>2014-08-07 09:55:18,790 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] START, ResumeBrokerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 1c5c5f8a
>2014-08-07 09:55:18,844 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] FINISH, ResumeBrokerVDSCommand, log id: 1c5c5f8a
>2014-08-07 09:55:18,845 INFO  [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] FINISH, ResumeVDSCommand, return: PoweringUp, log id: 626248c8
>2014-08-07 09:55:18,846 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM
>, sharedLocks= ]
>2014-08-07 09:55:18,908 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Correlation ID: 59a10b9a, Job ID: e5f550ac-cbe6-4c36-b9dd-0d8f7f5a84f5, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was resumed by admin (Host: rack).
>2014-08-07 09:55:21,273 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Paused --> Up
>2014-08-07 09:55:21,302 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: 59a10b9a, Job ID: e5f550ac-cbe6-4c36-b9dd-0d8f7f5a84f5, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack
>2014-08-07 09:55:30,768 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-08-07 09:55:30,960 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: d412eca
>2014-08-07 09:55:30,962 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: d412eca
>2014-08-07 09:55:31,063 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Running command: RunVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-08-07 09:55:31,119 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@cd426e7), log id: 7f00aeb
>2014-08-07 09:55:31,133 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7f00aeb
>2014-08-07 09:55:31,170 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: d879f9e
>2014-08-07 09:55:31,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 6b44275
>2014-08-07 09:55:31,311 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=ae661712-04dc-40c7-bb5d-ac71c44882b6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cb=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=710595e4-0cbc-4ae3-ae8d-3c123dad65cb, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901a=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=069719e3-08a8-492f-9e06-1d26df82901a, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cbdevice_1ea1dd78-6851-4585-8184-ea9b5bd84156=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=1ea1dd78-6851-4585-8184-ea9b5bd84156, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, index=0, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true
>2014-08-07 09:55:31,362 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, CreateVDSCommand, log id: 6b44275
>2014-08-07 09:55:31,374 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: d879f9e
>2014-08-07 09:55:31,374 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-08-07 09:55:31,387 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Correlation ID: 4c3d5594, Job ID: d0b854c7-90ae-4139-b7eb-cc2985d28c61, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack).
>2014-08-07 09:55:33,421 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp
>2014-08-07 09:55:33,426 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-36) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: f34781f
>2014-08-07 09:55:33,434 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-36) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=ae661712-04dc-40c7-bb5d-ac71c44882b6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cb=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=710595e4-0cbc-4ae3-ae8d-3c123dad65cb, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901a=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=069719e3-08a8-492f-9e06-1d26df82901a, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cbdevice_1ea1dd78-6851-4585-8184-ea9b5bd84156=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=1ea1dd78-6851-4585-8184-ea9b5bd84156, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=8661, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@3dba4305, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: f34781f
>2014-08-07 09:55:33,453 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
>2014-08-07 09:55:33,454 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5903, type=graphics}
>2014-08-07 09:55:38,935 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] Running command: StopVmCommand internal: false. Entities affected :  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-08-07 09:55:38,943 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 5984d7be
>2014-08-07 09:55:38,953 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 2ca735c0
>2014-08-07 09:55:39,449 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] FINISH, DestroyVDSCommand, log id: 2ca735c0
>2014-08-07 09:55:39,457 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] FINISH, DestroyVmVDSCommand, return: Down, log id: 5984d7be
>2014-08-07 09:55:39,465 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-16) [797ae1f9] Running command: ProcessDownVmCommand internal: true.
>2014-08-07 09:55:39,483 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] Correlation ID: 7b2c4cab, Job ID: 040e177f-3ede-4d2f-b2f9-6ce8e4c173b5, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest powered off by admin (Host: rack) (Reason: Not Specified).
>2014-08-07 10:17:39,711 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 10:17:39,716 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 10:30:54,990 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized
>2014-08-07 10:30:55,773 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-08-07 10:52:55,123 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 10:52:55,131 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 10:56:37,775 INFO  [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-9) [579bab3a] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 10:56:37,808 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [579bab3a] Correlation ID: 579bab3a, Call Stack: null, Custom Event ID: -1, Message: Data Center local, Compatibility Version 3.5 and Quota Type DISABLED was added by admin
>2014-08-07 10:56:52,943 INFO  [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-9) [3cf6d2dc] Running command: AddVdsGroupCommand internal: false. Entities affected :  ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool
>2014-08-07 10:56:52,990 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [3cf6d2dc] Correlation ID: 3cf6d2dc, Call Stack: null, Custom Event ID: -1, Message: Cluster local was added by admin
>2014-08-07 10:57:34,620 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-07 10:57:34,694 WARN  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,CANNOT_MAINTENANCE_VDS_RUN_VMS_NO_OTHER_RUNNING_VDS,$ClustersList Default,$HostsList rack
>2014-08-07 10:57:34,696 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-07 10:57:51,443 INFO  [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] Running command: StopVmCommand internal: false. Entities affected :  ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM
>2014-08-07 10:57:51,448 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 402666e2
>2014-08-07 10:57:51,455 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 15e94ca6
>2014-08-07 10:57:51,954 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] FINISH, DestroyVDSCommand, log id: 15e94ca6
>2014-08-07 10:57:51,962 INFO  [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] FINISH, DestroyVmVDSCommand, return: Down, log id: 402666e2
>2014-08-07 10:57:51,969 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-39) [2b8e1344] Running command: ProcessDownVmCommand internal: true.
>2014-08-07 10:57:51,987 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [20541247] Correlation ID: 20541247, Job ID: fb472bec-4098-4884-a42f-3071da83f9e8, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified).
>2014-08-07 10:58:17,194 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [4306be4f] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-07 10:58:17,227 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:17,232 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 801e5d2
>2014-08-07 10:58:17,236 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] VDS rack is spm and moved from up calling resetIrs.
>2014-08-07 10:58:17,239 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 7c3bbc08
>2014-08-07 10:58:17,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 294002ca
>2014-08-07 10:58:17,252 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-07 10:58:17,316 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, SpmStopVDSCommand, log id: 294002ca
>2014-08-07 10:58:17,322 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, ResetIrsVDSCommand, log id: 7c3bbc08
>2014-08-07 10:58:17,329 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, SetVdsStatusVDSCommand, log id: 801e5d2
>2014-08-07 10:58:17,390 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-07 10:58:17,426 INFO  [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Running command: MaintenanceVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:17,437 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Correlation ID: 4306be4f, Job ID: 03ca9fbb-39d3-4fd3-a560-9d66574f77e1, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin.
>2014-08-07 10:58:17,971 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack
>2014-08-07 10:58:17,994 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack.
>2014-08-07 10:58:17,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache
>2014-08-07 10:58:17,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache
>2014-08-07 10:58:18,001 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: 63f7f7c9
>2014-08-07 10:58:18,335 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : no reporting hosts
>2014-08-07 10:58:18,339 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts
>2014-08-07 10:58:18,345 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [43323ecc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 10:58:18,354 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [43323ecc] Correlation ID: 43323ecc, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 10:58:22,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, DisconnectStoragePoolVDSCommand, log id: 63f7f7c9
>2014-08-07 10:58:22,101 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 10:58:22,126 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 78657e38
>2014-08-07 10:58:22,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 78657e38
>2014-08-07 10:58:41,520 INFO  [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-10) [5447dae2] Running command: ChangeVDSClusterCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS,  ID: e14eab06-127a-4eaf-9220-ee1fd18c619e Type: VdsGroups
>2014-08-07 10:58:41,643 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 79d1c4e3
>2014-08-07 10:58:41,644 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] vdsManager::disposing
>2014-08-07 10:58:41,648 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] FINISH, RemoveVdsVDSCommand, log id: 79d1c4e3
>2014-08-07 10:58:41,652 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1e9829cf
>2014-08-07 10:58:41,653 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b
>2014-08-07 10:58:41,656 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager
>2014-08-07 10:58:41,657 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] Entered VdsManager constructor
>2014-08-07 10:58:41,689 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 10:58:41,692 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-10) [5447dae2] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 10:58:41,693 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] FINISH, AddVdsVDSCommand, log id: 1e9829cf
>2014-08-07 10:58:41,701 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-10) [5f80601f] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:41,724 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-10) [5f80601f] Lock freed to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: REGISTER_VDS
>, sharedLocks= ]
>2014-08-07 10:58:41,755 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5f80601f] Correlation ID: 5447dae2, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin.
>2014-08-07 10:58:41,804 INFO  [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] Running command: PersistentSetupNetworksCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:41,852 INFO  [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] Running command: SetupNetworksCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:41,855 INFO  [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] No changes were detected in setup networks for host rack (ID: 17f7fe56-5f32-40ef-b098-611b84484e0b)
>2014-08-07 10:58:41,941 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 10:58:41,942 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 10:58:41,949 INFO  [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [4d6f003d] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:41,977 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 10:58:41,978 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 10:58:41,985 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4d6f003d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 10:58:42,005 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4d6f003d] Correlation ID: 4d6f003d, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin.
>2014-08-07 10:58:44,692 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Initializing Host: rack
>2014-08-07 10:58:45,398 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-5) [7800e596] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-07 10:58:45,423 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:45,425 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Before acquiring lock in order to prevent monitoring for host rack from data-center local
>2014-08-07 10:58:45,426 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center local
>2014-08-07 10:58:45,436 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4cdc5d3b
>2014-08-07 10:58:45,443 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] FINISH, SetVdsStatusVDSCommand, log id: 4cdc5d3b
>2014-08-07 10:58:45,452 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Activate finished. Lock released. Monitoring can run now for host rack from data-center local
>2014-08-07 10:58:45,458 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [7800e596] Correlation ID: 7800e596, Job ID: 59ef2e2f-a80f-4cf7-ba48-c9c5dc27d587, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin.
>2014-08-07 10:58:45,463 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-07 10:58:47,783 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6a55d48c
>2014-08-07 10:58:47,839 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 6a55d48c
>2014-08-07 10:58:47,867 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 10:58:47,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 663e95a8
>2014-08-07 10:58:47,970 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] FINISH, GetHardwareInfoVDSCommand, log id: 663e95a8
>2014-08-07 10:58:48,037 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool
>2014-08-07 10:58:48,048 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15d9180e
>2014-08-07 10:58:48,141 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 15d9180e
>2014-08-07 10:58:48,148 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [553563f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 10:58:48,154 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [553563f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 10:58:48,182 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [2899c446] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 11:09:23,517 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 11:09:23,523 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:09:24,379 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [a701299] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:09:24,388 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3ada5403] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:09:24,396 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3ada5403] Correlation ID: 3ada5403, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:09:24,532 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 11:09:24,548 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [5e8f34a1] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:09:24,553 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [45264bb0] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:09:24,558 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [45264bb0] Correlation ID: 45264bb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:09:45,579 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 11:09:45,584 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:09:45,714 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [6f689abc] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:09:45,721 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [7900944a] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:09:45,727 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [7900944a] Correlation ID: 7900944a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:09:45,824 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 11:09:45,871 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [456e296a] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:09:45,879 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7e41eb29] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:09:45,883 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7e41eb29] Correlation ID: 7e41eb29, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:10:01,860 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 11:10:01,864 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:10:02,663 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1827f148] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:10:02,667 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [71147093] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:10:02,671 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71147093] Correlation ID: 71147093, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:10:15,016 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 11:10:15,019 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:10:15,047 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [746ec960] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:10:15,051 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [658768b3] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:10:15,055 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [658768b3] Correlation ID: 658768b3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:10:22,514 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 11:10:22,520 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:10:22,582 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [52d114e3] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:10:22,588 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [1853ef19] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:10:22,594 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [1853ef19] Correlation ID: 1853ef19, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:11:07,493 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 11:11:07,497 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:11:07,537 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2edb5230] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:11:07,546 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [6dc84935] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:11:07,550 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [6dc84935] Correlation ID: 6dc84935, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:11:39,905 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 11:11:39,909 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:11:39,927 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [519319d3] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:11:39,929 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4efbd73e] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:11:39,933 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4efbd73e] Correlation ID: 4efbd73e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:12:43,923 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 11:12:43,931 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:12:44,001 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:12:44,030 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:12:44,043 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7afdab99
>2014-08-07 11:12:44,072 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 7afdab99
>2014-08-07 11:12:44,081 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3c536aa6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.
>2014-08-07 11:12:44,096 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-6) [3c536aa6] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage.
>verify permission settings on the specified storage path.
>2014-08-07 11:12:44,107 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469)
>2014-08-07 11:12:44,122 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand.
>2014-08-07 11:12:44,134 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:12:44,143 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp--127.0.0.1-8702-6) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 11:12:44,157 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-6) Operation Failed: [Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.]
>2014-08-07 11:12:44,737 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [2ec5bc22] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:12:44,777 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [2b5631f5] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:12:44,785 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [2b5631f5] Correlation ID: 2b5631f5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:13:16,389 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 11:13:16,393 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:13:16,412 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:16,426 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:13:16,433 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [5e67349b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a4b8c71
>2014-08-07 11:13:16,467 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [5e67349b] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 3a4b8c71
>2014-08-07 11:13:16,475 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5e67349b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage returned by VDSM was: General Exception
>2014-08-07 11:13:16,483 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-10) [5e67349b] The connection with details /storage failed because of error code 100 and error message is: general exception
>2014-08-07 11:13:16,488 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: GeneralException (Failed with error GeneralException and code 100)
>2014-08-07 11:13:16,495 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand.
>2014-08-07 11:13:16,503 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Lock freed to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:16,508 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [General Exception]
>2014-08-07 11:13:16,530 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [23db14df] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:13:16,535 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [6a7821b4] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:13:16,541 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [6a7821b4] Correlation ID: 6a7821b4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:13:30,423 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 11:13:30,438 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:13:30,473 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:30,492 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:13:30,502 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 324196b4
>2014-08-07 11:13:30,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 324196b4
>2014-08-07 11:13:30,530 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [7d3ffe71] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.
>2014-08-07 11:13:30,544 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-8) [7d3ffe71] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage.
>verify permission settings on the specified storage path.
>2014-08-07 11:13:30,555 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469)
>2014-08-07 11:13:30,568 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand.
>2014-08-07 11:13:30,579 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:30,587 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-8) Operation Failed: [Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.]
>2014-08-07 11:13:30,631 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2d7ea28a] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:13:30,639 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [bd799c6] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:13:30,647 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [bd799c6] Correlation ID: bd799c6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:13:55,419 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 11:13:55,427 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:13:55,453 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:55,468 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:13:55,473 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d426297
>2014-08-07 11:13:55,489 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 5d426297
>2014-08-07 11:13:55,497 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [5663f6d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.
>2014-08-07 11:13:55,503 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-7) [5663f6d7] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage.
>verify permission settings on the specified storage path.
>2014-08-07 11:13:55,509 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469)
>2014-08-07 11:13:55,516 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand.
>2014-08-07 11:13:55,524 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:13:55,528 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Permission settings on the specified path do not allow access to the storage.
>Verify permission settings on the specified storage path.]
>2014-08-07 11:13:55,550 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [248841de] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:13:55,555 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [668eb07d] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:13:55,562 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [668eb07d] Correlation ID: 668eb07d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:14:04,931 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-07 11:14:04,938 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:14:04,998 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:14:05,023 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:14:05,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6ab0955e
>2014-08-07 11:14:05,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6ab0955e
>2014-08-07 11:14:05,093 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Lock freed to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION
>, sharedLocks= ]
>2014-08-07 11:14:05,151 INFO  [org.ovirt.engine.core.bll.storage.AddLocalStorageDomainCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] Running command: AddLocalStorageDomainCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:14:05,186 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 144338b2
>2014-08-07 11:14:05,207 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 144338b2
>2014-08-07 11:14:05,211 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@84831e5f, args=/storage), log id: 393be476
>2014-08-07 11:15:05,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, CreateStorageDomainVDSCommand, log id: 393be476
>2014-08-07 11:15:05,696 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=37f3c12d-f0b0-4916-8e01-9a290cd82ed0), log id: 11c193a7
>2014-08-07 11:15:05,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@6063117b, log id: 11c193a7
>2014-08-07 11:15:05,818 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-1) [8735d5] Running command: AttachStorageDomainToPoolCommand internal: true. Entities affected :  ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage
>2014-08-07 11:15:05,832 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [67d68dad] Lock Acquired to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL
>, sharedLocks= ]
>2014-08-07 11:15:05,863 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [67d68dad] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected :  ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool
>2014-08-07 11:15:06,005 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:15:06,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 79d3b6ea
>2014-08-07 11:15:06,039 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 79d3b6ea
>2014-08-07 11:15:06,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] START, CreateStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=29cf5221-8594-4e8b-afd3-bb0227304f01, storagePoolName=local, masterDomainId=37f3c12d-f0b0-4916-8e01-9a290cd82ed0, domainsIdList=[37f3c12d-f0b0-4916-8e01-9a290cd82ed0], masterVersion=1), log id: 261cb48c
>2014-08-07 11:16:06,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] FINISH, CreateStoragePoolVDSCommand, log id: 261cb48c
>2014-08-07 11:16:06,435 INFO  [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] Lock freed to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL
>, sharedLocks= ]
>2014-08-07 11:16:06,440 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>, sharedLocks= ]
>2014-08-07 11:16:06,450 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage
>2014-08-07 11:16:06,462 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>, sharedLocks= ]
>2014-08-07 11:16:06,469 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/7/14 11:16 AM
>2014-08-07 11:16:06,477 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:16:06,482 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 233421ba
>2014-08-07 11:16:06,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 233421ba
>2014-08-07 11:16:06,506 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [4b64550f] START, ActivateStorageDomainVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, storageDomainId = 37f3c12d-f0b0-4916-8e01-9a290cd82ed0), log id: 44f218e1
>2014-08-07 11:16:06,518 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool
>2014-08-07 11:16:06,541 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 4cca6b4f, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-07 11:16:06,558 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool local
>2014-08-07 11:16:06,575 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, masterVersion = 1), log id: 66cc42e7
>2014-08-07 11:17:06,822 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, ConnectStoragePoolVDSCommand, log id: 66cc42e7
>2014-08-07 11:17:06,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool local
>2014-08-07 11:17:06,874 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] starting spm on vds rack, storage pool local, prevId -1, LVER -1
>2014-08-07 11:17:06,883 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10aa41bf
>2014-08-07 11:17:06,943 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling started: taskId = 7c4cd73d-477c-467f-a023-2e127d2de394
>2014-08-07 11:17:07,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling ended: taskId = 7c4cd73d-477c-467f-a023-2e127d2de394 task status = finished
>2014-08-07 11:17:07,999 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling ended, spm status: SPM
>2014-08-07 11:17:08,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7c4cd73d-477c-467f-a023-2e127d2de394), log id: afcb23c
>2014-08-07 11:17:08,051 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, HSMClearTaskVDSCommand, log id: afcb23c
>2014-08-07 11:17:08,057 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@474618fa, log id: 10aa41bf
>2014-08-07 11:17:08,068 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 11:17:08,081 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 11:17:08,149 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, ActivateStorageDomainVDSCommand, log id: 44f218e1
>2014-08-07 11:17:08,152 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] ActivateStorage Domain. After Connect all hosts to pool. Time:8/7/14 11:17 AM
>2014-08-07 11:17:08,171 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, masterVersion = 1), log id: 5b07c499
>2014-08-07 11:18:08,388 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 5b07c499
>2014-08-07 11:18:08,390 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] ActivateStorage Domain. After change storage pool status in vds. Time:8/7/14 11:18 AM
>2014-08-07 11:18:08,401 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 4b64550f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd (Data Center local) was activated by <UNKNOWN>
>2014-08-07 11:18:08,421 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 67d68dad, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center local by <UNKNOWN>
>2014-08-07 11:18:08,437 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 8735d5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd was attached to Data Center local by <UNKNOWN>
>2014-08-07 11:18:08,466 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 5ca7e5d1, Job ID: 61d312f1-ada1-4d3b-9ee4-4f1bf0f8b6f9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd was added by admin
>2014-08-07 11:18:09,281 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [47a99ab4] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:18:09,286 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [4bcd6fb9] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:18:09,291 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4bcd6fb9] Correlation ID: 4bcd6fb9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:18:19,370 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 11:18:19,374 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 11:18:19,397 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1dc9b31c] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 11:18:19,408 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [2c7b6724] Running command: LogoutUserCommand internal: false.
>2014-08-07 11:18:19,419 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2c7b6724] Correlation ID: 2c7b6724, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 11:20:32,722 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-7) [6e2c45bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL
>, sharedLocks= ]
>2014-08-07 11:20:32,812 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage
>2014-08-07 11:20:32,832 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, DeactivateStorageDomainVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, storageDomainId = 37f3c12d-f0b0-4916-8e01-9a290cd82ed0, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 4561ad89
>2014-08-07 11:20:32,842 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, DeactivateStorageDomainVDSCommand, log id: 4561ad89
>2014-08-07 11:20:32,844 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL
>, sharedLocks= ]
>2014-08-07 11:20:32,846 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, SpmStopOnIrsVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false), log id: 5990381
>2014-08-07 11:20:32,848 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, ResetIrsVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 72806f4b
>2014-08-07 11:20:32,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01), log id: 6b22d588
>2014-08-07 11:20:32,859 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 29cf5221-8594-4e8b-afd3-bb0227304f01
>2014-08-07 11:20:32,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, SpmStopVDSCommand, log id: 6b22d588
>2014-08-07 11:20:32,913 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, ResetIrsVDSCommand, log id: 72806f4b
>2014-08-07 11:20:32,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, SpmStopOnIrsVDSCommand, log id: 5990381
>2014-08-07 11:20:32,920 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 4113d88d
>2014-08-07 11:20:32,972 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, DisconnectStoragePoolVDSCommand, log id: 4113d88d
>2014-08-07 11:20:32,975 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStoragePoolAsyncOperation] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] Disconnect storage pool treatment vds: rack,pool local
>2014-08-07 11:20:32,976 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 5b306cb2
>2014-08-07 11:20:33,022 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] FINISH, DisconnectStoragePoolVDSCommand, log id: 5b306cb2
>2014-08-07 11:20:33,029 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-4) After deactivate treatment vds: rack,pool local
>2014-08-07 11:20:33,034 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:20:33,038 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 567f46a6
>2014-08-07 11:20:33,188 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] FINISH, DisconnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 567f46a6
>2014-08-07 11:20:33,192 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 11:20:33,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7aa7c118
>2014-08-07 11:20:33,340 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] FINISH, DisconnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 7aa7c118
>2014-08-07 11:20:33,357 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] Correlation ID: 6e2c45bf, Job ID: 826fd762-3cfd-4894-a497-7c145b346c2a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd (Data Center local) was deactivated by admin
>2014-08-07 11:30:54,997 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized
>2014-08-07 11:30:55,806 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [2899c446] Attempting to update VMs/Templates Ovf.
>2014-08-07 12:17:31,446 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>, sharedLocks= ]
>2014-08-07 12:17:31,458 WARN  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] CanDoAction of action RemoveStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__REMOVE,ERROR_CANNOT_DETACH_LAST_STORAGE_DOMAIN
>2014-08-07 12:17:31,459 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE
>, sharedLocks= ]
>2014-08-07 12:17:45,475 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:17:45,480 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 424f728f
>2014-08-07 12:17:45,486 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] FINISH, SetVdsStatusVDSCommand, log id: 424f728f
>2014-08-07 12:17:45,567 INFO  [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] Running command: MaintenanceVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:17:45,576 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [3900667c] Correlation ID: 3900667c, Job ID: 1c7a7ce7-9366-4a76-bfa1-1c3cc8080fa6, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin.
>2014-08-07 12:17:45,845 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack
>2014-08-07 12:17:45,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Clearing cache of pool: 29cf5221-8594-4e8b-afd3-bb0227304f01 for problematic entities of VDS: rack.
>2014-08-07 12:17:45,870 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache
>2014-08-07 12:17:45,871 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache
>2014-08-07 12:17:45,875 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 69413136
>2014-08-07 12:17:45,921 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, DisconnectStoragePoolVDSCommand, log id: 69413136
>2014-08-07 12:17:45,926 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [8b21b6c] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool
>2014-08-07 12:18:01,090 INFO  [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-2) [5d35448e] Running command: ChangeVDSClusterCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS,  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups
>2014-08-07 12:18:01,100 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 30a79b77
>2014-08-07 12:18:01,101 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] vdsManager::disposing
>2014-08-07 12:18:01,102 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] FINISH, RemoveVdsVDSCommand, log id: 30a79b77
>2014-08-07 12:18:01,105 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e3cd999
>2014-08-07 12:18:01,106 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b
>2014-08-07 12:18:01,108 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager
>2014-08-07 12:18:01,109 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] Entered VdsManager constructor
>2014-08-07 12:18:01,130 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 12:18:01,134 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-2) [5d35448e] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 12:18:01,136 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] FINISH, AddVdsVDSCommand, log id: 4e3cd999
>2014-08-07 12:18:01,154 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-2) [2f7b25df] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:18:01,166 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-2) [2f7b25df] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: REGISTER_VDS
>, sharedLocks= ]
>2014-08-07 12:18:01,197 INFO  [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] Running command: PersistentSetupNetworksCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:18:01,208 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [2f7b25df] Correlation ID: 5d35448e, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin.
>2014-08-07 12:18:01,247 INFO  [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] Running command: SetupNetworksCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:18:01,248 INFO  [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] No changes were detected in setup networks for host rack (ID: 17f7fe56-5f32-40ef-b098-611b84484e0b)
>2014-08-07 12:18:04,136 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) [126267c5] Initializing Host: rack
>2014-08-07 12:18:06,465 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 12:18:06,466 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 12:18:06,467 INFO  [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [6f7ae589] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 12:18:06,481 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 12:18:06,482 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-07 12:18:06,489 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f7ae589] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 12:18:06,496 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f7ae589] Correlation ID: 6f7ae589, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin.
>2014-08-07 12:30:55,010 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized
>2014-08-07 12:30:55,810 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf.
>2014-08-07 13:13:25,248 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-11) [69a5df17] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-07 13:13:25,274 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 13:13:25,275 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Before acquiring lock in order to prevent monitoring for host rack from data-center Default
>2014-08-07 13:13:25,276 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default
>2014-08-07 13:13:25,282 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6079145b
>2014-08-07 13:13:25,291 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] FINISH, SetVdsStatusVDSCommand, log id: 6079145b
>2014-08-07 13:13:25,348 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default
>2014-08-07 13:13:25,377 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Correlation ID: 69a5df17, Job ID: 0d45ef1f-8908-4172-b6d3-c2759d8fa2ec, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin.
>2014-08-07 13:13:25,389 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-07 13:13:28,292 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 42c08c8e
>2014-08-07 13:13:28,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GetHardwareInfoVDSCommand, log id: 42c08c8e
>2014-08-07 13:13:28,381 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 13:13:28,424 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 31aaebd6
>2014-08-07 13:13:28,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] FINISH, GetHardwareInfoVDSCommand, log id: 31aaebd6
>2014-08-07 13:13:28,533 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-16) [3194387f] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 13:13:28,542 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 13:13:28,553 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7443bab0
>2014-08-07 13:13:28,703 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7443bab0
>2014-08-07 13:13:28,706 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Host rack storage connection was succeeded 
>2014-08-07 13:13:28,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 186a7137
>2014-08-07 13:13:52,837 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-08-07 13:13:52,844 WARN  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] CanDoAction of action ActivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-07 13:13:52,846 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-08-07 13:14:29,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 186a7137
>2014-08-07 13:14:29,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 33fb4bec
>2014-08-07 13:14:29,345 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 33fb4bec
>2014-08-07 13:14:29,358 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 13:14:29,364 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 13:14:29,381 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [53385366] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 13:14:29,486 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 13:14:29,492 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 13:14:29,495 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d9ba7e5
>2014-08-07 13:14:29,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling started: taskId = 60a9bf0d-9f4c-4468-b4c3-547edf64be16
>2014-08-07 13:14:30,551 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling ended: taskId = 60a9bf0d-9f4c-4468-b4c3-547edf64be16 task status = finished
>2014-08-07 13:14:30,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling ended, spm status: SPM
>2014-08-07 13:14:30,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=60a9bf0d-9f4c-4468-b4c3-547edf64be16), log id: 7c1c6b59
>2014-08-07 13:14:30,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, HSMClearTaskVDSCommand, log id: 7c1c6b59
>2014-08-07 13:14:30,654 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a8b3539, log id: 5d9ba7e5
>2014-08-07 13:14:30,658 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 13:14:30,665 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 13:14:30,678 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 57693b23
>2014-08-07 13:14:30,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 13:14:30,708 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ba620ff
>2014-08-07 13:14:30,713 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ba620ff
>2014-08-07 13:14:30,714 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 57693b23
>2014-08-07 13:14:30,715 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default
>2014-08-07 13:14:54,761 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-08-07 13:14:54,780 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 13:14:54,784 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e1e952a
>2014-08-07 13:17:04,848 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [656265fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN.
>2014-08-07 13:17:04,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 7e1e952a
>2014-08-07 13:17:04,852 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [656265fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target
>2014-08-07 13:17:04,854 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-41) [656265fc] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target
>2014-08-07 13:17:04,859 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-07 13:17:04,861 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 2b849a32
>2014-08-07 13:19:05,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Failed in AttachStorageDomainVDS method
>2014-08-07 13:19:05,342 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358
>2014-08-07 13:19:05,346 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] FINISH, AttachStorageDomainVDSCommand, log id: 2b849a32
>2014-08-07 13:19:05,348 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
>2014-08-07 13:19:05,373 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Command [id=84a2ceb4-886a-4665-a8fb-56449a88a69a]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 13:19:05,393 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Correlation ID: 76d232c7, Job ID: 09bed423-fc5d-4cae-a270-aeceba1112d1, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin)
>2014-08-07 13:30:55,017 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized
>2014-08-07 13:30:55,814 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
>2014-08-07 13:51:49,118 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 13:51:49,126 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:53:38,697 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 13:53:38,702 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:54:15,483 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 13:54:15,487 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:57:04,806 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 13:57:04,815 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:57:19,925 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 13:57:19,930 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:58:06,010 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 13:58:06,013 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:58:20,611 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 13:58:20,615 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 13:58:21,820 INFO  [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Lock Acquired to object EngineLock [exclusiveLocks= key: newVM value: VM_NAME
>key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-08-07 13:58:21,900 INFO  [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Running command: AddVmFromSnapshotCommand internal: false. Entities affected :  ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups,  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage,  ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM
>2014-08-07 13:58:21,962 INFO  [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Locking VM(id = 10a64f7a-b891-46c7-be45-36a8d2b0a754) with compensation.
>2014-08-07 13:58:21,965 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [1209e408] START, SetVmStatusVDSCommand( vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, status = ImageLocked, exit status = Normal), log id: 2fa884d3
>2014-08-07 13:58:21,974 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [1209e408] FINISH, SetVmStatusVDSCommand, log id: 2fa884d3
>2014-08-07 13:58:21,978 INFO  [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Lock freed to object EngineLock [exclusiveLocks= key: newVM value: VM_NAME
>key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM
>, sharedLocks= ]
>2014-08-07 13:58:22,001 INFO  [org.ovirt.engine.core.bll.CopyImageGroupCommand] (ajp--127.0.0.1-8702-6) [41881ba1] Running command: CopyImageGroupCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 13:58:22,285 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, dstImageGroupId = 348a2b24-96f9-4d7c-93bd-771f95224b0b, vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, dstImageId = d0fd3f56-e0a1-495c-b41b-dfa272854235, imageDescription = , dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 337b3ddf
>2014-08-07 13:58:22,291 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID
>2014-08-07 13:58:22,292 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] -- copyImage parameters:
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                vmGUID=10a64f7a-b891-46c7-be45-36a8d2b0a754
>                srcImageGUID=6cae69b6-2d4f-4e61-993c-dce113dd19e5
>                srcVolUUID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b
>                dstImageGUID=348a2b24-96f9-4d7c-93bd-771f95224b0b
>                dstVolUUID=d0fd3f56-e0a1-495c-b41b-dfa272854235
>                descr=
>                dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-07 13:58:22,744 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 337b3ddf
>2014-08-07 13:58:22,813 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-6) [41881ba1] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command f99f4bac-e57f-4749-b26e-751c2a3b26e7
>2014-08-07 13:58:22,814 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-6) [41881ba1] CommandMultiAsyncTasks::AttachTask: Attaching task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb to command f99f4bac-e57f-4749-b26e-751c2a3b26e7.
>2014-08-07 13:58:22,850 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-6) [41881ba1] Adding task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-08-07 13:58:22,982 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [41881ba1] Correlation ID: 1209e408, Job ID: 44e11b7d-d743-4d68-95f6-d691deaf5aa1, Call Stack: null, Custom Event ID: -1, Message: VM newVM creation was initiated by admin.
>2014-08-07 13:58:22,983 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-6) [41881ba1] BaseAsyncTask::startPollingTask: Starting to poll task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb.
>2014-08-07 13:58:28,125 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-57) [699d5b35] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-08-07 13:58:28,133 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-57) [699d5b35] SPMAsyncTask::PollTask: Polling task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-08-07 13:58:28,135 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-57) [699d5b35] Finished polling Tasks, will poll again in 10 seconds.
>2014-08-07 14:00:21,368 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [699d5b35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack has network interface which exceeded the defined threshold [95%] (em1: transmit rate[0%], receive rate [98%])
>2014-08-07 14:06:04,423 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-73) SPMAsyncTask::PollTask: Polling task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-08-07 14:06:04,435 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-73) BaseAsyncTask::OnTaskEndSuccess: Task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-08-07 14:06:04,437 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-73) CommandAsyncTask::EndActionIfNecessary: All tasks of command f99f4bac-e57f-4749-b26e-751c2a3b26e7 has ended -> executing endAction
>2014-08-07 14:06:04,439 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-73) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: f99f4bac-e57f-4749-b26e-751c2a3b26e7): calling endAction .
>2014-08-07 14:06:04,442 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmFromSnapshot, executionIndex: 0
>2014-08-07 14:06:04,526 INFO  [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (org.ovirt.thread.pool-8-thread-37) [1209e408] Ending command successfully: org.ovirt.engine.core.bll.AddVmFromSnapshotCommand
>2014-08-07 14:06:04,636 INFO  [org.ovirt.engine.core.bll.CopyImageGroupCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] Ending command successfully: org.ovirt.engine.core.bll.CopyImageGroupCommand
>2014-08-07 14:06:04,644 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 348a2b24-96f9-4d7c-93bd-771f95224b0b, imageId = d0fd3f56-e0a1-495c-b41b-dfa272854235), log id: 5019143b
>2014-08-07 14:06:04,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@de298e57, log id: 5019143b
>2014-08-07 14:06:04,698 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, SetVmStatusVDSCommand( vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, status = Down, exit status = Normal), log id: 3f71eefb
>2014-08-07 14:06:04,706 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, SetVmStatusVDSCommand, log id: 3f71eefb
>2014-08-07 14:06:04,756 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) Correlation ID: 1209e408, Call Stack: null, Custom Event ID: -1, Message: VM newVM creation has been completed.
>2014-08-07 14:06:04,757 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromSnapshot completed, handling the result.
>2014-08-07 14:06:04,758 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromSnapshot succeeded, clearing tasks.
>2014-08-07 14:06:04,764 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb
>2014-08-07 14:06:04,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb), log id: 1e39fb86
>2014-08-07 14:06:04,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7b1882c9-f22a-4ede-b8fc-ad390dfe51bb), log id: 70b029ea
>2014-08-07 14:06:04,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, HSMClearTaskVDSCommand, log id: 70b029ea
>2014-08-07 14:06:04,821 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, SPMClearTaskVDSCommand, log id: 1e39fb86
>2014-08-07 14:06:04,838 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) BaseAsyncTask::RemoveTaskFromDB: Removed task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb from DataBase
>2014-08-07 14:06:04,841 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity f99f4bac-e57f-4749-b26e-751c2a3b26e7
>2014-08-07 14:07:25,906 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) Setting new tasks map. The map contains now 0 tasks
>2014-08-07 14:07:25,909 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-08-07 14:09:27,243 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack
>2014-08-07 14:09:27,246 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack
>2014-08-07 14:10:55,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 14:10:55,903 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:10:55,906 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) [301effc7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:10:55,909 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) [301effc7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:10:55,924 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [301effc7] Correlation ID: 301effc7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain).
>2014-08-07 14:12:55,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 14:12:55,953 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) [301effc7] hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 14:12:55,954 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] IrsBroker::Failed::GetStoragePoolInfoVDS
>2014-08-07 14:12:55,955 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',)
>2014-08-07 14:12:55,991 INFO  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 14:12:56,010 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4a63088b
>2014-08-07 14:12:56,013 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 30b01c9
>2014-08-07 14:12:56,019 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:14:55,921 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
>2014-08-07 14:14:55,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-87) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 14:14:55,933 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond
>2014-08-07 14:14:55,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null!
>2014-08-07 14:14:55,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] FINISH, SpmStopVDSCommand, log id: 30b01c9
>2014-08-07 14:14:55,934 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-07 14:14:55,942 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy83.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-07 14:14:55,952 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] FINISH, ResetIrsVDSCommand, log id: 4a63088b
>2014-08-07 14:14:55,990 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-07 14:14:55,993 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] Correlation ID: d1e91b4, Job ID: 5525121d-f1d6-4b7b-b002-594242ccff24, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default.
>2014-08-07 14:14:56,004 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [301effc7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 14:14:56,006 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35
>2014-08-07 14:14:56,071 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now.
>2014-08-07 14:14:56,089 WARN  [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [ce514ae] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
>2014-08-07 14:14:56,096 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) [ce514ae] Removing vds [] from the domain in maintenance cache
>2014-08-07 14:14:56,097 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) [ce514ae] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 14:14:56,098 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [301effc7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue
>2014-08-07 14:14:56,099 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export
>2014-08-07 14:14:56,104 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational.
>2014-08-07 14:14:56,121 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 14:14:56,129 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 14:14:56,135 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-07 14:14:56,145 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Correlation ID: ecf806c, Job ID: 457b2c2b-2326-4352-980c-9e4808604c03, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
>2014-08-07 14:14:56,151 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Removing vds [] from the domain in maintenance cache
>2014-08-07 14:14:56,152 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Removing host(s) [] from hosts unseen domain report cache
>2014-08-07 14:14:59,606 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4a33b6d9
>2014-08-07 14:14:59,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 4a33b6d9
>2014-08-07 14:14:59,688 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [3f132663] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:14:59,740 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [4151a43a] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:14:59,748 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:14:59,761 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e4d6360
>2014-08-07 14:14:59,827 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e4d6360
>2014-08-07 14:14:59,828 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] Host rack storage connection was succeeded 
>2014-08-07 14:14:59,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 352a3b66
>2014-08-07 14:15:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering 2 storage domains
>2014-08-07 14:15:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-07 14:15:00,009 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 14:15:00,010 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:15 PM
>2014-08-07 14:15:00,016 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:15 PM
>2014-08-07 14:15:00,018 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [39f70177] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987
>2014-08-07 14:15:00,021 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] Running command: ConnectDomainToStorageCommand internal: true. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-07 14:15:00,022 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:15 PM
>2014-08-07 14:15:00,032 INFO  [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:15 PM
>2014-08-07 14:15:05,943 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 14:15:05,947 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts
>2014-08-07 14:16:00,268 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, ConnectStoragePoolVDSCommand, log id: 352a3b66
>2014-08-07 14:16:00,282 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d5be635
>2014-08-07 14:16:00,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d5be635
>2014-08-07 14:16:00,383 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [598afe80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 14:16:00,389 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [598afe80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 14:16:00,407 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [2512845a] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:16:06,094 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 14:16:06,100 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 14:16:06,102 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2de7461b
>2014-08-07 14:16:06,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling started: taskId = 08a4a8e1-a14e-4900-a42a-117a0fbfab0a
>2014-08-07 14:16:07,157 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended: taskId = 08a4a8e1-a14e-4900-a42a-117a0fbfab0a task status = finished
>2014-08-07 14:16:07,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended, spm status: SPM
>2014-08-07 14:16:07,219 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=08a4a8e1-a14e-4900-a42a-117a0fbfab0a), log id: 22b925d0
>2014-08-07 14:16:07,264 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, HSMClearTaskVDSCommand, log id: 22b925d0
>2014-08-07 14:16:07,265 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c22f65e, log id: 2de7461b
>2014-08-07 14:16:07,273 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:16:07,279 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:16:07,308 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3a9705a8
>2014-08-07 14:16:07,330 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:16:07,334 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 31455495
>2014-08-07 14:16:07,341 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31455495
>2014-08-07 14:16:07,343 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a9705a8
>2014-08-07 14:16:07,344 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-39) Discovered no tasks on Storage Pool Default
>2014-08-07 14:27:11,031 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:27:11,031 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:27:11,032 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:27:11,033 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:27:11,034 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:27:11,034 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:27:11,035 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:27:11,036 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:27:11,036 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:27:11,037 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:27:11,040 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:27:11,040 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:27:11,041 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:27:11,041 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:27:11,042 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:27:11,042 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:27:11,043 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:27:11,043 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:27:11,044 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:27:11,044 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:27:11,044 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:27:11,045 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:27:11,046 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:27:11,046 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:27:11,046 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:27:11,047 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:27:11,048 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:27:11,049 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:27:11,049 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:27:11,050 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:27:11,050 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:27:11,050 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:27:11,051 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:27:11,051 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:27:11,052 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:27:11,052 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:27:11,053 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:27:11,053 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:27:11,054 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:27:11,054 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:27:11,055 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:27:11,055 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:27:11,056 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:27:11,056 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:27:11,057 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:27:11,057 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:27:11,058 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:27:11,058 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:27:11,059 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:27:11,059 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:27:11,060 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:27:11,060 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:27:11,061 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:27:11,061 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:27:11,062 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:27:11,062 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:27:11,063 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:27:11,063 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:27:11,064 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:27:11,064 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:27:11,065 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:27:11,065 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:27:11,065 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:27:11,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:27:11,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:27:11,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:27:11,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:27:11,068 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:27:11,996 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend
>2014-08-07 14:27:12,799 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:27:12,936 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:27:12,937 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries
>2014-08-07 14:27:12,941 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries
>2014-08-07 14:27:12,941 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager
>2014-08-07 14:27:12,942 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:27:12,987 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager
>2014-08-07 14:27:12,988 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager
>2014-08-07 14:27:12,988 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:27:12,990 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:27:12,992 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector
>2014-08-07 14:27:13,001 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree
>2014-08-07 14:27:13,014 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector
>2014-08-07 14:27:13,014 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:27:13,023 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:27:13,188 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers
>2014-08-07 14:27:13,431 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:27:13,477 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:27:13,480 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:27:13,480 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:27:13,481 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:27:13,482 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:27:13,534 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:27:13,536 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:27:13,537 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager
>2014-08-07 14:27:13,541 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager
>2014-08-07 14:27:13,546 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector
>2014-08-07 14:27:13,553 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector
>2014-08-07 14:27:13,617 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list
>2014-08-07 14:27:13,618 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:27:13,619 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:27:13,620 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list
>2014-08-07 14:27:13,789 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-4) Start initializing DbUserCacheManager
>2014-08-07 14:27:13,805 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-4) Finished initializing DbUserCacheManager
>2014-08-07 14:27:13,847 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-4) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:27:13,852 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) Start initializing ResourceManager
>2014-08-07 14:27:13,889 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:27:13,950 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-4) Entered VdsManager constructor
>2014-08-07 14:27:14,016 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-4) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:27:14,361 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:27:14,451 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) Finished initializing ResourceManager
>2014-08-07 14:27:14,464 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-4) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:27:14,465 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Start scheduling to enable vds load balancer
>2014-08-07 14:27:14,471 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Finished scheduling to enable vds load balancer
>2014-08-07 14:27:14,472 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Start HA Reservation check
>2014-08-07 14:27:14,474 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Finished HA Reservation check
>2014-08-07 14:27:14,486 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-08-07 14:27:14,504 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-4) Init VM custom properties utilities
>2014-08-07 14:27:14,505 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-4) Init device custom properties utilities
>2014-08-07 14:27:14,514 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:27:14,520 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Initializing Scheduling manager
>2014-08-07 14:27:14,591 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) External scheduler disabled, discovery skipped
>2014-08-07 14:27:14,591 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Initialized Scheduling manager
>2014-08-07 14:27:14,592 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-4) Initializing DWH Heart Beat
>2014-08-07 14:27:14,608 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-4) DWH Heart Beat initialized
>2014-08-07 14:27:15,079 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 14:27:15,084 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:27:15,088 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:27:15,089 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:27:15,089 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:27:15,090 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:27:15,090 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:27:15,091 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:27:15,092 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:27:15,092 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:27:15,093 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:27:15,094 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:27:15,094 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:27:15,095 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:27:15,095 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:27:15,096 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:27:15,097 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:27:15,097 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:27:15,098 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:27:15,300 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:27:18,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: b5b109f
>2014-08-07 14:27:18,247 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: b5b109f
>2014-08-07 14:27:18,510 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:27:18,532 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [6dc144af] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:27:22,298 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 14:27:22,309 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:27:24,522 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:27:24,573 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:27:24,585 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:27:24,756 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Correlation ID: 74f60fa4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:27:24,983 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [74f60fa4] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:27:25,007 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:27:25,071 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:27:25,125 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: a458c7d
>2014-08-07 14:27:25,181 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:27:25,193 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6d5cd523
>2014-08-07 14:27:25,206 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d5cd523
>2014-08-07 14:27:25,208 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a458c7d
>2014-08-07 14:27:25,209 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] Discovered no tasks on Storage Pool Default
>2014-08-07 14:27:40,203 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 14:27:40,223 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:28:52,393 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:28:52,394 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:28:52,396 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:28:52,397 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:28:52,417 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:28:52,418 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:28:52,419 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:28:52,419 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:28:52,420 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:28:52,420 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:28:52,426 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:28:52,427 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:28:52,427 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:28:52,428 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:28:52,428 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:28:52,429 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:28:52,429 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:28:52,430 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:28:52,430 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:28:52,432 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:28:52,432 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:28:52,433 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:28:52,434 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:28:52,434 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:28:52,435 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:28:52,436 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:28:52,436 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:28:52,437 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:28:52,437 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:28:52,438 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:28:52,438 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:28:52,439 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:28:52,439 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:28:52,440 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:28:52,440 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:28:52,441 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:28:52,441 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:28:52,443 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:28:52,443 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:28:52,444 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:28:52,444 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:28:52,445 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:28:52,446 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:28:52,446 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:28:52,446 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:28:52,447 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:28:52,448 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:28:52,448 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:28:52,449 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:28:52,450 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:28:52,451 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:28:52,451 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:28:52,452 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:28:52,453 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:28:52,453 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:28:52,453 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:28:52,454 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:28:52,454 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:28:52,455 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:28:52,455 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:28:52,456 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:28:52,456 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:28:52,456 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:28:52,457 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:28:52,457 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:28:52,458 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:28:52,459 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:28:52,459 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:28:53,219 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend
>2014-08-07 14:28:53,623 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:28:53,712 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:28:53,712 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries
>2014-08-07 14:28:53,715 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries
>2014-08-07 14:28:53,715 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager
>2014-08-07 14:28:53,716 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:28:53,736 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager
>2014-08-07 14:28:53,737 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager
>2014-08-07 14:28:53,737 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:28:53,740 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:28:53,742 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector
>2014-08-07 14:28:53,751 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree
>2014-08-07 14:28:53,759 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector
>2014-08-07 14:28:53,759 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:28:53,766 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:28:53,835 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers
>2014-08-07 14:28:53,861 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:28:53,872 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:28:53,873 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:28:53,874 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:28:53,875 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:28:53,875 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:28:53,908 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:28:53,909 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:28:53,909 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager
>2014-08-07 14:28:53,912 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager
>2014-08-07 14:28:53,916 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector
>2014-08-07 14:28:53,922 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector
>2014-08-07 14:28:53,974 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list
>2014-08-07 14:28:53,975 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:28:53,976 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:28:53,977 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list
>2014-08-07 14:28:54,081 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager
>2014-08-07 14:28:54,082 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager
>2014-08-07 14:28:54,131 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:28:54,134 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager
>2014-08-07 14:28:54,173 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:28:54,282 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor
>2014-08-07 14:28:54,351 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:28:54,617 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:28:54,669 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager
>2014-08-07 14:28:54,678 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:28:54,679 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer
>2014-08-07 14:28:54,681 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer
>2014-08-07 14:28:54,682 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check
>2014-08-07 14:28:54,683 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check
>2014-08-07 14:28:54,711 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-08-07 14:28:54,716 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities
>2014-08-07 14:28:54,717 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities
>2014-08-07 14:28:54,728 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager
>2014-08-07 14:28:54,745 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:28:54,797 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped
>2014-08-07 14:28:54,798 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager
>2014-08-07 14:28:54,798 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat
>2014-08-07 14:28:54,803 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized
>2014-08-07 14:28:56,030 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 14:28:56,039 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:28:56,049 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:28:56,050 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:28:56,051 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:28:56,052 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:28:56,053 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:28:56,054 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:28:56,055 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:28:56,056 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:28:56,057 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:28:56,058 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:28:56,059 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:28:56,060 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:28:56,061 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:28:56,062 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:28:56,064 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:28:56,065 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:28:56,067 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:28:56,340 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:28:58,515 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 69864cda
>2014-08-07 14:28:58,570 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 69864cda
>2014-08-07 14:28:58,857 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:28:58,878 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [45bfd8e8] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:29:04,743 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [93c60b5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:29:04,754 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [93c60b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:29:04,769 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [93c60b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:29:04,814 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [93c60b5] Correlation ID: 93c60b5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:29:04,987 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [93c60b5] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:29:05,005 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [93c60b5] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:29:05,027 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [93c60b5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:29:05,114 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f502e86
>2014-08-07 14:29:05,172 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:29:05,185 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5da22ddc
>2014-08-07 14:29:05,196 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5da22ddc
>2014-08-07 14:29:05,198 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f502e86
>2014-08-07 14:29:05,199 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [93c60b5] Discovered no tasks on Storage Pool Default
>2014-08-07 14:29:22,110 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 14:29:22,115 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:29:40,201 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 14:29:40,220 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:31:40,853 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:31:40,853 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:31:40,854 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:31:40,855 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:31:41,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:31:41,066 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:31:41,067 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:31:41,068 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:31:41,068 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:31:41,069 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:31:41,069 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:31:41,077 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:31:41,077 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:31:41,078 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:31:41,078 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:31:41,079 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:31:41,079 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:31:41,080 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:31:41,080 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:31:41,081 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:31:41,081 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:31:41,082 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:31:41,082 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:31:41,092 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:31:41,093 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:31:41,093 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:31:41,094 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:31:41,094 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:31:41,095 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:31:41,095 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:31:41,096 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:31:41,096 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:31:41,096 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:31:41,097 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:31:41,097 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:31:41,100 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:31:41,101 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:31:41,101 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:31:41,102 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:31:41,103 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:31:41,126 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:31:41,126 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:31:41,127 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:31:41,127 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:31:41,128 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:31:41,139 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:31:41,140 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:31:41,140 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:31:41,141 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:31:41,141 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:31:41,157 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:31:41,157 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:31:41,158 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:31:41,158 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:31:41,159 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:31:41,159 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:31:41,160 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:31:41,160 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:31:41,161 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:31:41,167 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:31:41,168 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:31:41,168 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:31:41,168 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:31:41,169 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:31:41,169 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:31:41,170 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:31:41,170 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:31:41,186 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:31:41,587 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null
>2014-08-07 14:31:42,174 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend
>2014-08-07 14:31:42,243 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null
>2014-08-07 14:31:43,295 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null
>2014-08-07 14:31:43,305 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:31:43,311 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null
>2014-08-07 14:31:43,332 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null
>2014-08-07 14:31:43,351 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null
>2014-08-07 14:31:43,377 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null
>2014-08-07 14:31:43,399 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null
>2014-08-07 14:31:43,418 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null
>2014-08-07 14:31:43,444 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null
>2014-08-07 14:31:43,463 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null
>2014-08-07 14:31:43,482 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null
>2014-08-07 14:31:43,487 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:31:43,488 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries
>2014-08-07 14:31:43,494 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries
>2014-08-07 14:31:43,494 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager
>2014-08-07 14:31:43,495 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:31:43,505 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null
>2014-08-07 14:31:43,536 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null
>2014-08-07 14:31:43,548 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager
>2014-08-07 14:31:43,548 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager
>2014-08-07 14:31:43,549 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:31:43,554 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:31:43,555 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null
>2014-08-07 14:31:43,559 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector
>2014-08-07 14:31:43,581 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null
>2014-08-07 14:31:43,582 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree
>2014-08-07 14:31:43,596 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector
>2014-08-07 14:31:43,596 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:31:43,607 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null
>2014-08-07 14:31:43,608 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:31:43,634 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null
>2014-08-07 14:31:43,652 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null
>2014-08-07 14:31:43,678 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null
>2014-08-07 14:31:43,698 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null
>2014-08-07 14:31:43,717 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null
>2014-08-07 14:31:43,736 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null
>2014-08-07 14:31:43,761 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null
>2014-08-07 14:31:43,779 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null
>2014-08-07 14:31:43,794 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers
>2014-08-07 14:31:43,799 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null
>2014-08-07 14:31:43,819 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null
>2014-08-07 14:31:43,846 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:31:43,850 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null
>2014-08-07 14:31:43,872 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null
>2014-08-07 14:31:43,878 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:31:43,879 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:31:43,880 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:31:43,887 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:31:43,888 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:31:43,898 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null
>2014-08-07 14:31:43,921 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null
>2014-08-07 14:31:43,944 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null
>2014-08-07 14:31:43,962 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:31:43,962 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null
>2014-08-07 14:31:43,962 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:31:43,964 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager
>2014-08-07 14:31:43,967 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager
>2014-08-07 14:31:43,969 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector
>2014-08-07 14:31:43,978 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector
>2014-08-07 14:31:43,982 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null
>2014-08-07 14:31:44,007 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null
>2014-08-07 14:31:44,026 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null
>2014-08-07 14:31:44,045 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null
>2014-08-07 14:31:44,071 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null
>2014-08-07 14:31:44,079 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list
>2014-08-07 14:31:44,080 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:31:44,081 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:31:44,082 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list
>2014-08-07 14:31:44,230 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager
>2014-08-07 14:31:44,245 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager
>2014-08-07 14:31:44,268 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:31:44,276 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager
>2014-08-07 14:31:44,365 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:31:44,436 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor
>2014-08-07 14:31:44,487 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 14:31:44,507 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:31:44,512 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:31:44,529 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:31:44,529 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:31:44,533 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:31:44,533 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:31:44,534 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:31:44,535 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:31:44,535 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:31:44,545 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:31:44,548 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:31:44,548 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:31:44,549 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:31:44,549 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:31:44,550 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:31:44,554 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:31:44,555 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:31:44,556 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:31:44,556 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:31:44,828 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:31:44,891 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager
>2014-08-07 14:31:44,904 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:31:44,905 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer
>2014-08-07 14:31:44,912 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer
>2014-08-07 14:31:44,913 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check
>2014-08-07 14:31:44,917 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check
>2014-08-07 14:31:44,946 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities
>2014-08-07 14:31:44,950 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities
>2014-08-07 14:31:44,952 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges
>2014-08-07 14:31:44,986 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager
>2014-08-07 14:31:44,995 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:31:45,057 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped
>2014-08-07 14:31:45,058 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager
>2014-08-07 14:31:45,059 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat
>2014-08-07 14:31:45,061 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized
>2014-08-07 14:31:45,210 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:31:45,558 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 14:31:45,595 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [2e5e0418] Running command: LogoutUserCommand internal: false.
>2014-08-07 14:31:45,634 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [2e5e0418] Correlation ID: 2e5e0418, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 14:31:46,050 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 14:31:48,803 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5122ceb2
>2014-08-07 14:31:48,853 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 5122ceb2
>2014-08-07 14:31:49,138 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:31:49,157 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [64e470f4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:31:54,958 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:31:54,981 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:31:54,996 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:31:55,041 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Correlation ID: 737e3ec1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:31:55,216 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [737e3ec1] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:31:55,235 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:31:55,276 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:31:55,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39c28664
>2014-08-07 14:31:55,436 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:31:55,459 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6fce3e95
>2014-08-07 14:31:55,516 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6fce3e95
>2014-08-07 14:31:55,517 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39c28664
>2014-08-07 14:31:55,518 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] Discovered no tasks on Storage Pool Default
>2014-08-07 14:36:16,507 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:36:16,518 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:36:16,519 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:36:16,520 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:36:16,534 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:36:16,535 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:36:16,545 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:36:16,546 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:36:16,547 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:36:16,556 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:36:16,557 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:36:16,558 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:36:16,559 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:36:16,574 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:36:16,574 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:36:16,575 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:36:16,576 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:36:16,576 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:36:16,581 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:36:16,582 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:36:16,583 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:36:16,583 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:36:16,584 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:36:16,593 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:36:16,594 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:36:16,595 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:36:16,595 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:36:16,596 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:36:16,602 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:36:16,603 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:36:16,604 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:36:16,604 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:36:16,605 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:36:16,606 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:36:16,606 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:36:16,607 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:36:16,620 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:36:16,621 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:36:16,632 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:36:16,632 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:36:16,633 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:36:16,634 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:36:16,634 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:36:16,638 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:36:16,638 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:36:16,639 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:36:16,640 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:36:16,650 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:36:16,651 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:36:16,652 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:36:16,653 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:36:16,653 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:36:16,666 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:36:16,666 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:36:16,667 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:36:16,668 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:36:16,669 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:36:16,669 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:36:16,670 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:36:16,671 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:36:16,672 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:36:16,685 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:36:16,685 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:36:16,686 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:36:16,687 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:36:16,688 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:36:16,688 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:36:16,689 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:36:17,675 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend
>2014-08-07 14:36:18,295 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:36:18,418 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:36:18,418 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries
>2014-08-07 14:36:18,421 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries
>2014-08-07 14:36:18,422 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager
>2014-08-07 14:36:18,422 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:36:18,465 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager
>2014-08-07 14:36:18,466 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager
>2014-08-07 14:36:18,466 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:36:18,474 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:36:18,476 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector
>2014-08-07 14:36:18,490 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree
>2014-08-07 14:36:18,510 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector
>2014-08-07 14:36:18,511 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:36:18,528 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:36:18,693 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers
>2014-08-07 14:36:18,748 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-08-07 14:36:18,914 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:36:18,916 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:36:18,916 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-08-07 14:36:18,917 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:36:18,918 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:36:18,973 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:36:18,974 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:36:18,975 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager
>2014-08-07 14:36:18,980 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager
>2014-08-07 14:36:18,982 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector
>2014-08-07 14:36:18,994 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector
>2014-08-07 14:36:19,077 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list
>2014-08-07 14:36:19,080 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:36:19,085 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:36:19,086 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list
>2014-08-07 14:36:19,197 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager
>2014-08-07 14:36:19,202 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager
>2014-08-07 14:36:19,244 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:36:19,254 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager
>2014-08-07 14:36:19,306 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:36:19,387 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor
>2014-08-07 14:36:19,448 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:36:19,730 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:36:19,804 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager
>2014-08-07 14:36:19,820 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:36:19,822 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer
>2014-08-07 14:36:19,823 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer
>2014-08-07 14:36:19,824 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check
>2014-08-07 14:36:19,830 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check
>2014-08-07 14:36:19,841 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
>2014-08-07 14:36:19,855 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities
>2014-08-07 14:36:19,856 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities
>2014-08-07 14:36:19,867 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager
>2014-08-07 14:36:19,874 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:36:19,951 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped
>2014-08-07 14:36:19,952 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager
>2014-08-07 14:36:19,952 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat
>2014-08-07 14:36:19,962 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized
>2014-08-07 14:36:20,619 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 14:36:20,625 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:36:20,628 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:36:20,629 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:36:20,630 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:36:20,630 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:36:20,632 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:36:20,632 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:36:20,633 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:36:20,634 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:36:20,638 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:36:20,639 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:36:20,639 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:36:20,640 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:36:20,646 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:36:20,647 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:36:20,647 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:36:20,648 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:36:20,648 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:36:20,844 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:36:23,646 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4f7ba324
>2014-08-07 14:36:23,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 4f7ba324
>2014-08-07 14:36:23,974 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:36:24,000 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [326de99b] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:36:29,939 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [63b0815] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:36:29,960 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [63b0815] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:36:29,967 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [63b0815] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:36:30,040 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [63b0815] Correlation ID: 63b0815, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:36:30,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [63b0815] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:36:30,258 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [63b0815] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:36:30,293 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [63b0815] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:36:30,348 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 506d2be2
>2014-08-07 14:36:30,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:36:30,474 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6f4d7b0b
>2014-08-07 14:36:30,488 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f4d7b0b
>2014-08-07 14:36:30,490 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 506d2be2
>2014-08-07 14:36:30,491 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [63b0815] Discovered no tasks on Storage Pool Default
>2014-08-07 14:36:45,894 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:36:45,895 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:36:45,895 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:36:45,896 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:36:45,897 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:36:45,897 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:36:45,898 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:36:45,898 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:36:45,899 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:36:45,899 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:36:45,904 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:36:45,905 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:36:45,905 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:36:45,905 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:36:45,906 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:36:45,906 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:36:45,907 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:36:45,907 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:36:45,908 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:36:45,908 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:36:45,908 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:36:45,909 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:36:45,909 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:36:45,910 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:36:45,910 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:36:45,911 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:36:45,911 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:36:45,912 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:36:45,912 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:36:45,913 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:36:45,930 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:36:45,931 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:36:45,931 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:36:45,932 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:36:45,932 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:36:45,932 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:36:45,933 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:36:45,933 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:36:45,934 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:36:45,934 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:36:45,935 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:36:45,935 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:36:45,936 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:36:45,936 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:36:45,937 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:36:45,937 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:36:45,938 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:36:45,938 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:36:45,939 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:36:45,939 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:36:45,940 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:36:45,942 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:36:45,943 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:36:45,943 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:36:45,944 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:36:45,944 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:36:45,944 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:36:45,945 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:36:45,945 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:36:45,946 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:36:45,946 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:36:45,947 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:36:45,947 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:36:45,949 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:36:45,949 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:36:45,953 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:36:45,953 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:36:45,954 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:36:46,575 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend
>2014-08-07 14:36:47,147 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:36:47,273 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:36:47,274 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries
>2014-08-07 14:36:47,278 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries
>2014-08-07 14:36:47,279 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager
>2014-08-07 14:36:47,280 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:36:47,312 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager
>2014-08-07 14:36:47,313 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager
>2014-08-07 14:36:47,314 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:36:47,321 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:36:47,324 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector
>2014-08-07 14:36:47,342 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree
>2014-08-07 14:36:47,353 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector
>2014-08-07 14:36:47,354 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:36:47,364 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:36:47,456 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers
>2014-08-07 14:36:47,493 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:36:47,505 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:36:47,507 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:36:47,507 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl
>2014-08-07 14:36:47,508 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:36:47,509 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:36:47,544 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:36:47,545 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:36:47,546 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager
>2014-08-07 14:36:47,547 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager
>2014-08-07 14:36:47,550 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector
>2014-08-07 14:36:47,553 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector
>2014-08-07 14:36:47,601 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list
>2014-08-07 14:36:47,602 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:36:47,602 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:36:47,604 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list
>2014-08-07 14:36:47,723 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager
>2014-08-07 14:36:47,733 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager
>2014-08-07 14:36:47,747 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:36:47,749 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager
>2014-08-07 14:36:47,779 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:36:47,842 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Entered VdsManager constructor
>2014-08-07 14:36:47,908 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:36:47,925 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 14:36:47,939 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:36:47,947 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:36:47,947 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:36:47,948 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:36:47,948 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:36:47,958 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:36:47,959 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:36:47,959 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:36:47,960 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:36:47,961 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:36:47,968 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:36:47,969 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:36:47,970 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:36:47,970 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:36:47,971 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:36:47,971 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:36:47,972 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:36:47,980 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:36:48,244 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:36:48,287 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager
>2014-08-07 14:36:48,295 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:36:48,295 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer
>2014-08-07 14:36:48,297 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer
>2014-08-07 14:36:48,297 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check
>2014-08-07 14:36:48,298 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check
>2014-08-07 14:36:48,307 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges
>2014-08-07 14:36:48,316 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities
>2014-08-07 14:36:48,318 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities
>2014-08-07 14:36:48,330 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager
>2014-08-07 14:36:48,343 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:36:48,392 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped
>2014-08-07 14:36:48,393 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager
>2014-08-07 14:36:48,394 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat
>2014-08-07 14:36:48,401 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized
>2014-08-07 14:36:48,460 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:36:52,076 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 32a6111c
>2014-08-07 14:36:52,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 32a6111c
>2014-08-07 14:36:52,434 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:36:52,453 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [22a9866e] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:36:58,429 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [6e21a806] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:36:58,444 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [6e21a806] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:36:58,453 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [6e21a806] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:36:58,569 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6e21a806] Correlation ID: 6e21a806, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:36:58,746 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [6e21a806] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:36:58,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [6e21a806] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:36:58,791 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6e21a806] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:36:58,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6e3d4086
>2014-08-07 14:36:58,921 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:36:58,945 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 657ce6b0
>2014-08-07 14:36:58,956 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 657ce6b0
>2014-08-07 14:36:58,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e3d4086
>2014-08-07 14:36:58,959 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [6e21a806] Discovered no tasks on Storage Pool Default
>2014-08-07 14:37:22,290 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:37:22,291 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:37:22,292 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:37:22,292 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:37:22,293 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:37:22,300 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:37:22,302 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:37:22,303 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:37:22,303 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:37:22,304 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:37:22,305 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:37:22,305 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:37:22,306 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:37:22,306 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:37:22,307 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:37:22,313 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:37:22,314 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:37:22,315 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:37:22,315 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:37:22,315 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:37:22,316 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:37:22,316 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:37:22,317 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:37:22,317 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:37:22,318 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:37:22,318 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:37:22,319 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:37:22,323 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:37:22,323 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:37:22,324 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:37:22,324 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:37:22,325 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:37:22,325 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:37:22,326 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:37:22,327 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:37:22,327 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:37:22,334 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:37:22,336 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:37:22,337 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:37:22,337 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:37:22,337 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:37:22,338 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:37:22,338 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:37:22,339 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:37:22,339 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:37:22,340 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:37:22,340 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:37:22,341 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:37:22,348 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:37:22,349 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:37:22,350 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:37:22,350 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:37:22,351 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:37:22,351 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:37:22,352 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:37:22,352 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:37:22,353 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:37:22,353 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:37:22,353 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:37:22,355 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:37:22,359 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:37:22,359 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:37:22,360 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:37:22,360 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:37:22,366 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:37:22,367 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:37:22,367 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:37:22,368 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:37:23,197 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend
>2014-08-07 14:37:23,858 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:37:23,927 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null
>2014-08-07 14:37:24,031 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:37:24,032 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries
>2014-08-07 14:37:24,037 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries
>2014-08-07 14:37:24,038 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager
>2014-08-07 14:37:24,040 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:37:24,063 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null
>2014-08-07 14:37:24,096 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager
>2014-08-07 14:37:24,097 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager
>2014-08-07 14:37:24,098 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:37:24,108 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:37:24,110 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector
>2014-08-07 14:37:24,128 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree
>2014-08-07 14:37:24,146 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector
>2014-08-07 14:37:24,147 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:37:24,159 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:37:24,195 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null
>2014-08-07 14:37:24,293 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers
>2014-08-07 14:37:24,325 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null
>2014-08-07 14:37:24,351 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-08-07 14:37:24,375 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:37:24,376 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:37:24,377 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl
>2014-08-07 14:37:24,378 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:37:24,379 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:37:24,420 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:37:24,422 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:37:24,423 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager
>2014-08-07 14:37:24,426 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager
>2014-08-07 14:37:24,451 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null
>2014-08-07 14:37:24,450 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector
>2014-08-07 14:37:24,455 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector
>2014-08-07 14:37:24,523 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list
>2014-08-07 14:37:24,524 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:37:24,526 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:37:24,527 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list
>2014-08-07 14:37:24,631 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager
>2014-08-07 14:37:24,645 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager
>2014-08-07 14:37:24,680 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:37:24,702 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager
>2014-08-07 14:37:24,716 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:37:24,819 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor
>2014-08-07 14:37:24,888 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:37:24,896 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 14:37:24,911 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:37:24,914 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:37:24,915 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:37:24,916 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:37:24,916 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:37:24,923 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:37:24,924 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:37:24,924 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:37:24,925 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:37:24,926 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:37:24,926 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:37:24,927 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:37:24,928 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:37:24,928 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:37:24,937 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:37:24,937 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:37:24,938 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:37:24,938 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:37:25,043 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:37:25,099 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager
>2014-08-07 14:37:25,116 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:37:25,117 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer
>2014-08-07 14:37:25,119 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer
>2014-08-07 14:37:25,119 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check
>2014-08-07 14:37:25,120 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check
>2014-08-07 14:37:25,129 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges
>2014-08-07 14:37:25,157 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities
>2014-08-07 14:37:25,160 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities
>2014-08-07 14:37:25,173 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager
>2014-08-07 14:37:25,175 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:37:25,230 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped
>2014-08-07 14:37:25,231 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager
>2014-08-07 14:37:25,232 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat
>2014-08-07 14:37:25,238 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized
>2014-08-07 14:37:25,414 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:37:28,846 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: b98d041
>2014-08-07 14:37:28,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: b98d041
>2014-08-07 14:37:29,180 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:37:29,203 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [44bd54c7] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:37:35,166 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:37:35,176 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:37:35,187 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:37:35,230 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Correlation ID: 32f66d2c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:37:35,378 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [32f66d2c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:37:35,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:37:35,419 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:37:35,485 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 644cfc0f
>2014-08-07 14:37:35,555 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:37:35,571 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2ffedd7f
>2014-08-07 14:37:35,581 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ffedd7f
>2014-08-07 14:37:35,582 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 644cfc0f
>2014-08-07 14:37:35,583 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] Discovered no tasks on Storage Pool Default
>2014-08-07 14:37:50,503 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
>2014-08-07 14:37:50,504 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
>2014-08-07 14:37:50,505 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
>2014-08-07 14:37:50,505 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
>2014-08-07 14:37:50,506 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
>2014-08-07 14:37:50,507 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
>2014-08-07 14:37:50,507 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true".
>2014-08-07 14:37:50,517 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702".
>2014-08-07 14:37:50,517 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
>2014-08-07 14:37:50,518 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
>2014-08-07 14:37:50,518 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
>2014-08-07 14:37:50,518 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
>2014-08-07 14:37:50,519 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine".
>2014-08-07 14:37:50,527 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
>2014-08-07 14:37:50,528 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost".
>2014-08-07 14:37:50,528 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
>2014-08-07 14:37:50,529 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
>2014-08-07 14:37:50,529 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***".
>2014-08-07 14:37:50,530 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432".
>2014-08-07 14:37:50,530 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False".
>2014-08-07 14:37:50,534 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
>2014-08-07 14:37:50,534 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
>2014-08-07 14:37:50,535 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine".
>2014-08-07 14:37:50,545 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "".
>2014-08-07 14:37:50,546 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
>2014-08-07 14:37:50,546 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
>2014-08-07 14:37:50,547 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
>2014-08-07 14:37:50,547 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com".
>2014-08-07 14:37:50,548 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt".
>2014-08-07 14:37:50,548 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g".
>2014-08-07 14:37:50,556 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g".
>2014-08-07 14:37:50,556 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false".
>2014-08-07 14:37:50,557 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None".
>2014-08-07 14:37:50,558 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
>2014-08-07 14:37:50,568 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false".
>2014-08-07 14:37:50,568 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None".
>2014-08-07 14:37:50,569 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
>2014-08-07 14:37:50,570 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
>2014-08-07 14:37:50,581 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
>2014-08-07 14:37:50,581 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
>2014-08-07 14:37:50,582 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
>2014-08-07 14:37:50,583 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m".
>2014-08-07 14:37:50,584 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m".
>2014-08-07 14:37:50,584 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
>2014-08-07 14:37:50,597 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
>2014-08-07 14:37:50,598 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
>2014-08-07 14:37:50,599 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
>2014-08-07 14:37:50,608 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
>2014-08-07 14:37:50,609 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
>2014-08-07 14:37:50,610 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
>2014-08-07 14:37:50,611 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
>2014-08-07 14:37:50,611 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
>2014-08-07 14:37:50,620 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true".
>2014-08-07 14:37:50,621 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
>2014-08-07 14:37:50,622 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
>2014-08-07 14:37:50,622 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml".
>2014-08-07 14:37:50,623 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1".
>2014-08-07 14:37:50,624 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10".
>2014-08-07 14:37:50,632 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
>2014-08-07 14:37:50,633 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
>2014-08-07 14:37:50,634 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine".
>2014-08-07 14:37:50,634 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt".
>2014-08-07 14:37:50,635 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
>2014-08-07 14:37:50,644 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
>2014-08-07 14:37:50,645 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false".
>2014-08-07 14:37:50,645 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
>2014-08-07 14:37:50,646 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
>2014-08-07 14:37:50,647 INFO  [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD".
>2014-08-07 14:37:51,468 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Start initializing Backend
>2014-08-07 14:37:51,911 INFO  [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-4) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
>2014-08-07 14:37:52,026 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6
>2014-08-07 14:37:52,027 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Start initializing dictionaries
>2014-08-07 14:37:52,034 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Finished initializing dictionaries
>2014-08-07 14:37:52,034 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing AuditLogCleanupManager
>2014-08-07 14:37:52,035 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting audit cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:37:52,056 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing AuditLogCleanupManager
>2014-08-07 14:37:52,057 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing CommandEntityCleanupManager
>2014-08-07 14:37:52,057 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting command entity cleanup manager to run at: 35 35 3 * * ?
>2014-08-07 14:37:52,062 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing CommandEntityCleanupManager
>2014-08-07 14:37:52,064 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Start initializing TagsDirector
>2014-08-07 14:37:52,085 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Tag root added to tree
>2014-08-07 14:37:52,104 INFO  [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Finished initializing TagsDirector
>2014-08-07 14:37:52,105 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Start initializing IsoDomainListSyncronizer
>2014-08-07 14:37:52,109 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null
>2014-08-07 14:37:52,117 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Finished initializing IsoDomainListSyncronizer
>2014-08-07 14:37:52,203 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Completed initializing handlers
>2014-08-07 14:37:52,230 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
>2014-08-07 14:37:52,250 WARN  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
>2014-08-07 14:37:52,251 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:37:52,253 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
>2014-08-07 14:37:52,254 INFO  [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
>2014-08-07 14:37:52,255 INFO  [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Mark incomplete jobs as UNKNOWN
>2014-08-07 14:37:52,295 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Start initializing JobRepositoryCleanupManager
>2014-08-07 14:37:52,296 INFO  [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Finished initializing JobRepositoryCleanupManager
>2014-08-07 14:37:52,296 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Start initializing AutoRecoveryManager
>2014-08-07 14:37:52,299 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Finished initializing AutoRecoveryManager
>2014-08-07 14:37:52,300 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Start initializing ExecutionMessageDirector
>2014-08-07 14:37:52,304 INFO  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Finished initializing ExecutionMessageDirector
>2014-08-07 14:37:52,364 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Start of enabled extensions list
>2014-08-07 14:37:52,364 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:37:52,365 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0',  File: 'N/A', Initialized: 'true'
>2014-08-07 14:37:52,367 INFO  [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) End of enabled extensions list
>2014-08-07 14:37:52,469 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager
>2014-08-07 14:37:52,472 INFO  [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager
>2014-08-07 14:37:52,520 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully.
>2014-08-07 14:37:52,522 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager
>2014-08-07 14:37:52,576 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-07 14:37:52,691 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor
>2014-08-07 14:37:52,772 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (10.34.62.208,54,321)
>2014-08-07 14:37:52,957 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 14:37:52,969 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types
>2014-08-07 14:37:52,975 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: UNASSIGNED not exist in string table
>2014-08-07 14:37:52,976 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table
>2014-08-07 14:37:52,976 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table
>2014-08-07 14:37:52,977 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table
>2014-08-07 14:37:52,978 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
>2014-08-07 14:37:52,980 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
>2014-08-07 14:37:52,981 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
>2014-08-07 14:37:52,982 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
>2014-08-07 14:37:52,983 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table
>2014-08-07 14:37:52,984 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
>2014-08-07 14:37:52,985 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_STOPPED not exist in string table
>2014-08-07 14:37:52,985 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_STARTED not exist in string table
>2014-08-07 14:37:52,986 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_ERROR not exist in string table
>2014-08-07 14:37:52,995 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table
>2014-08-07 14:37:52,996 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:37:52,997 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table
>2014-08-07 14:37:52,998 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table
>2014-08-07 14:37:53,108 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager
>2014-08-07 14:37:53,151 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager
>2014-08-07 14:37:53,162 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully.
>2014-08-07 14:37:53,163 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer
>2014-08-07 14:37:53,169 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer
>2014-08-07 14:37:53,170 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check
>2014-08-07 14:37:53,171 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check
>2014-08-07 14:37:53,180 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges
>2014-08-07 14:37:53,188 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities
>2014-08-07 14:37:53,189 INFO  [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities
>2014-08-07 14:37:53,197 INFO  [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252
>2014-08-07 14:37:53,204 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager
>2014-08-07 14:37:53,243 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped
>2014-08-07 14:37:53,244 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager
>2014-08-07 14:37:53,244 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat
>2014-08-07 14:37:53,257 INFO  [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized
>2014-08-07 14:37:53,322 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 14:37:56,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3e13b940
>2014-08-07 14:37:56,833 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 3e13b940
>2014-08-07 14:37:57,057 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack
>2014-08-07 14:37:57,080 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [46247d52] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 14:38:03,241 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [175991ea] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 14:38:03,264 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [175991ea] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null
>2014-08-07 14:38:03,271 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [175991ea] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null
>2014-08-07 14:38:03,345 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [175991ea] Correlation ID: 175991ea, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 14:38:03,526 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [175991ea] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-07 14:38:03,537 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [175991ea] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 14:38:03,555 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [175991ea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 14:38:03,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 52fc122e
>2014-08-07 14:38:03,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 14:38:03,685 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7d448d31
>2014-08-07 14:38:03,699 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d448d31
>2014-08-07 14:38:03,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52fc122e
>2014-08-07 14:38:03,702 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [175991ea] Discovered no tasks on Storage Pool Default
>2014-08-07 14:47:04,647 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-07 15:37:52,657 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized
>2014-08-07 15:37:53,163 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf.
>2014-08-07 15:49:57,126 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-07 15:49:57,133 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 15:49:57,155 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [fcd8d62] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:49:57,183 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [1615ec50] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:49:57,204 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [1615ec50] Correlation ID: 1615ec50, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 15:49:59,294 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 15:50:01,733 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [5b9c38ac] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:50:01,736 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [1b0b2960] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:50:01,740 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [1b0b2960] Correlation ID: 1b0b2960, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 15:54:30,563 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 15:54:30,570 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 15:54:30,582 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [262446e2] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:54:30,587 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [7e2285fd] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:54:30,596 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7e2285fd] Correlation ID: 7e2285fd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 15:54:32,230 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 15:54:32,798 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [633648f5] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:54:32,802 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [1b94efa] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:54:32,806 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1b94efa] Correlation ID: 1b94efa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 15:55:38,518 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 15:55:38,524 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 15:55:38,539 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [6862e83] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:55:38,545 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [69e17b3b] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:55:38,552 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [69e17b3b] Correlation ID: 69e17b3b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 15:55:39,899 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 15:55:40,328 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [77f231bd] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 15:55:40,332 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [78465d1d] Running command: LogoutUserCommand internal: false.
>2014-08-07 15:55:40,337 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [78465d1d] Correlation ID: 78465d1d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 16:04:52,004 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 16:04:52,012 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 16:04:53,815 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginAdminUserCommand internal: false.
>2014-08-07 16:04:59,089 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 16:04:59,094 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 16:05:32,225 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 16:05:32,229 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 16:05:32,238 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [50490546] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 16:05:32,247 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [ae2fc2] Running command: LogoutUserCommand internal: false.
>2014-08-07 16:05:32,257 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [ae2fc2] Correlation ID: ae2fc2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 16:05:34,047 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 16:05:34,392 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [3242db73] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 16:05:34,400 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [7bbf4404] Running command: LogoutUserCommand internal: false.
>2014-08-07 16:05:34,404 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [7bbf4404] Correlation ID: 7bbf4404, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 16:27:31,622 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 16:27:31,629 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 16:27:31,636 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [35895a] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 16:27:31,640 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [3098c960] Running command: LogoutUserCommand internal: false.
>2014-08-07 16:27:31,647 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [3098c960] Correlation ID: 3098c960, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 16:27:32,949 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 16:27:33,440 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [574a1f51] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 16:27:33,443 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [4a999e1e] Running command: LogoutUserCommand internal: false.
>2014-08-07 16:27:33,448 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [4a999e1e] Correlation ID: 4a999e1e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 16:37:52,669 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-07 16:37:53,592 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf.
>2014-08-07 16:57:20,044 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-08-07 16:57:20,140 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 16:57:20,150 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: caab5d3
>2014-08-07 16:59:30,285 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN.
>2014-08-07 16:59:30,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: caab5d3
>2014-08-07 16:59:30,291 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target
>2014-08-07 16:59:30,295 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-42) [2e959e56] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target
>2014-08-07 16:59:30,318 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand.
>2014-08-07 16:59:30,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 5dfbba80
>2014-08-07 17:01:30,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Failed in AttachStorageDomainVDS method
>2014-08-07 17:01:30,920 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358
>2014-08-07 17:01:30,935 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] FINISH, AttachStorageDomainVDSCommand, log id: 5dfbba80
>2014-08-07 17:01:30,939 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358)
>2014-08-07 17:01:30,994 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Command [id=cb7ecc9d-6a6a-41e9-bb28-b5694d83abb7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 17:01:31,031 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Correlation ID: 277d6f20, Job ID: d1972209-3011-4316-9e81-c90bf97a25f9, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin)
>2014-08-07 17:13:26,085 INFO  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-08-07 17:13:26,099 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:13:26,105 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 22ba5258
>2014-08-07 17:14:28,971 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0}, log id: 22ba5258
>2014-08-07 17:14:28,976 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 6814f7d7
>2014-08-07 17:14:43,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, AttachStorageDomainVDSCommand, log id: 6814f7d7
>2014-08-07 17:14:43,329 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 7601da2
>2014-08-07 17:14:43,436 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, GetImagesListVDSCommand, return: [11111111-1111-1111-1111-111111111111], log id: 7601da2
>2014-08-07 17:14:43,462 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, GetVolumesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933, imageGroupId = 11111111-1111-1111-1111-111111111111), log id: 7dd47f5b
>2014-08-07 17:14:43,512 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, GetVolumesListVDSCommand, return: [], log id: 7dd47f5b
>2014-08-07 17:14:43,513 ERROR [org.ovirt.engine.core.bll.GetUnregisteredDisksQuery] (org.ovirt.thread.pool-8-thread-21) [225a8704] Could not get populated disk, reason: null
>2014-08-07 17:14:43,515 WARN  [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-21) There are no OVF_STORE disks on storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933
>2014-08-07 17:14:43,523 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Lock Acquired to object EngineLock [exclusiveLocks= key: 63fe3516-4170-402a-9dc8-6d3eab1c7933 value: STORAGE
>, sharedLocks= ]
>2014-08-07 17:14:43,544 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Running command: ActivateStorageDomainCommand internal: true. Entities affected :  ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage
>2014-08-07 17:14:43,559 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Lock freed to object EngineLock [exclusiveLocks= key: 63fe3516-4170-402a-9dc8-6d3eab1c7933 value: STORAGE
>, sharedLocks= ]
>2014-08-07 17:14:43,561 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/7/14 5:14 PM
>2014-08-07 17:14:43,567 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:14:43,574 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 454862f3
>2014-08-07 17:14:43,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0}, log id: 454862f3
>2014-08-07 17:14:43,592 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 5f634ce9
>2014-08-07 17:14:43,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] FINISH, ActivateStorageDomainVDSCommand, log id: 5f634ce9
>2014-08-07 17:14:43,744 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. After Connect all hosts to pool. Time:8/7/14 5:14 PM
>2014-08-07 17:14:43,763 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2c46611a
>2014-08-07 17:14:44,088 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 2c46611a
>2014-08-07 17:14:44,089 INFO  [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-15) Refreshed vds rack in pool Default
>2014-08-07 17:14:44,090 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. After change storage pool status in vds. Time:8/7/14 5:14 PM
>2014-08-07 17:14:44,095 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4829e40a
>2014-08-07 17:14:44,131 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [948eec4] Correlation ID: 948eec4, Call Stack: null, Custom Event ID: -1, Message: Storage Domain ISO_DOMAIN (Data Center Default) was activated by <UNKNOWN>
>2014-08-07 17:14:44,139 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [948eec4] Correlation ID: 225a8704, Job ID: 71065b7b-b89f-46d4-82db-c8523e67889c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain ISO_DOMAIN was attached to Data Center Default by admin
>2014-08-07 17:14:44,156 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] FINISH, GetFileStatsVDSCommand, return: {}, log id: 4829e40a
>2014-08-07 17:14:44,167 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27bd6ccf
>2014-08-07 17:14:44,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] FINISH, GetFileStatsVDSCommand, return: {}, log id: 27bd6ccf
>2014-08-07 17:37:52,676 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized
>2014-08-07 17:37:53,636 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
>2014-08-07 17:40:58,795 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 17:40:58,801 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:41:00,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3df34cd7
>2014-08-07 17:41:00,643 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3df34cd7
>2014-08-07 17:41:00,651 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5e1c7be7
>2014-08-07 17:41:00,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetFileStatsVDSCommand, return: {}, log id: 5e1c7be7
>2014-08-07 17:41:00,707 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list succeeded for domain(s): ISO_DOMAIN (All file type)
>2014-08-07 17:45:32,882 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 17:45:32,889 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:45:32,955 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [74d3cf3] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:45:32,960 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [221af9ce] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:45:32,965 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [221af9ce] Correlation ID: 221af9ce, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:45:33,412 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 17:45:33,485 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2e06c8ab] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:45:33,490 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [8897b51] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:45:33,496 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [8897b51] Correlation ID: 8897b51, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:45:57,952 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 17:45:57,956 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:45:58,788 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [44406a47] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:45:58,791 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [11e60992] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:45:58,795 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [11e60992] Correlation ID: 11e60992, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:48:30,780 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 17:48:30,786 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:48:31,611 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [58001122] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:48:31,615 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [6f5d23b2] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:48:31,622 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f5d23b2] Correlation ID: 6f5d23b2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:48:51,350 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 17:48:51,352 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-43) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-07 17:48:51,354 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:48:51,394 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [58459177] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:48:51,397 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [36dcd673] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:48:51,406 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [36dcd673] Correlation ID: 36dcd673, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:49:15,882 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-7) [493d107a] The message key AddBookmark is missing from bundles/ExecutionMessages
>2014-08-07 17:49:15,916 INFO  [org.ovirt.engine.core.bll.AddBookmarkCommand] (ajp--127.0.0.1-8702-7) [493d107a] Running command: AddBookmarkCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:49:15,943 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [493d107a] Correlation ID: 493d107a, Job ID: dc802394-6944-4a31-8869-e2216492ff43, Call Stack: null, Custom Event ID: -1, Message: Bookmark test was added by admin.
>2014-08-07 17:49:25,251 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 17:49:25,256 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:49:25,266 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6ace6da3] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:49:25,270 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [43a1c18f] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:49:25,275 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [43a1c18f] Correlation ID: 43a1c18f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:50:01,784 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 17:50:01,795 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:50:03,150 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [1c6b0ab1] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:50:03,154 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [f349986] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:50:03,159 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [f349986] Correlation ID: f349986, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:50:22,035 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 17:50:22,049 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:50:22,080 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-11) [32e0de7] The message key AddBookmark is missing from bundles/ExecutionMessages
>2014-08-07 17:50:22,110 INFO  [org.ovirt.engine.core.bll.AddBookmarkCommand] (ajp--127.0.0.1-8702-11) [32e0de7] Running command: AddBookmarkCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:50:22,125 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [32e0de7] Correlation ID: 32e0de7, Job ID: 573c6899-eb68-4072-93f9-5cfe67db3be4, Call Stack: null, Custom Event ID: -1, Message: Bookmark foo was added by admin.
>2014-08-07 17:50:22,985 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [1a4b6286] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:50:22,988 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [5f1cea48] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:50:22,994 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [5f1cea48] Correlation ID: 5f1cea48, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:50:31,646 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 17:50:31,650 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:50:31,665 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [613224bf] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:50:31,668 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [8fe854e] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:50:31,677 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [8fe854e] Correlation ID: 8fe854e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:50:41,863 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 17:50:41,868 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:50:41,894 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-10) [2a4e5287] The message key RemoveBookmark is missing from bundles/ExecutionMessages
>2014-08-07 17:50:41,921 INFO  [org.ovirt.engine.core.bll.RemoveBookmarkCommand] (ajp--127.0.0.1-8702-10) [2a4e5287] Running command: RemoveBookmarkCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:50:41,945 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2a4e5287] Correlation ID: 2a4e5287, Job ID: cb44395a-ed6d-4fce-a592-18fb9f5fad68, Call Stack: null, Custom Event ID: -1, Message: Bookmark test was removed by admin.
>2014-08-07 17:50:42,677 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [2d7f9d31] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:50:42,681 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [66f6c478] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:50:42,684 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [66f6c478] Correlation ID: 66f6c478, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:50:58,098 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false.
>2014-08-07 17:50:58,102 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:50:58,117 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [10a292e2] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:50:58,126 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [61d7a27c] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:50:58,139 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [61d7a27c] Correlation ID: 61d7a27c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:51:42,594 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 17:51:42,598 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:51:42,624 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-6) [62803950] The message key UpdateBookmark is missing from bundles/ExecutionMessages
>2014-08-07 17:51:42,650 INFO  [org.ovirt.engine.core.bll.UpdateBookmarkCommand] (ajp--127.0.0.1-8702-6) [62803950] Running command: UpdateBookmarkCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 17:51:42,663 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [62803950] Correlation ID: 62803950, Job ID: 9341c107-be35-45a2-9e3b-d4ac4cbca465, Call Stack: null, Custom Event ID: -1, Message: Bookmark foo was updated by admin.
>2014-08-07 17:51:42,675 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [13c7ca57] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:51:42,680 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [17034c5] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:51:42,685 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [17034c5] Correlation ID: 17034c5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:52:04,607 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 17:52:04,612 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:52:04,618 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [753c9bb0] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:52:04,621 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [3972eaff] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:52:04,626 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [3972eaff] Correlation ID: 3972eaff, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:53:14,908 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 17:53:14,911 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:53:14,972 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [3d8faed7] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:53:14,977 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [6bd2b9a7] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:53:14,981 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [6bd2b9a7] Correlation ID: 6bd2b9a7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:54:13,534 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 17:54:13,539 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 17:54:14,348 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [102dcf63] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 17:54:14,352 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [6f61bf18] Running command: LogoutUserCommand internal: false.
>2014-08-07 17:54:14,356 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [6f61bf18] Correlation ID: 6f61bf18, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 17:54:50,572 INFO  [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp--127.0.0.1-8702-9) Running command: AddVmTemplateCommand internal: false. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 17:54:50,582 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-9) START, SetVmStatusVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, status = ImageLocked, exit status = Normal), log id: 3f908ee5
>2014-08-07 17:54:50,597 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, SetVmStatusVDSCommand, log id: 3f908ee5
>2014-08-07 17:54:50,652 INFO  [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (ajp--127.0.0.1-8702-9) [63c518d5] Running command: CreateImageTemplateCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 17:54:50,679 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614, dstImageGroupId = a5dac258-8f4f-48ae-b8a0-1e58a6b190db, vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, dstImageId = 46253726-8595-4414-b570-faf47fd84c4f, imageDescription = Active VM, dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 7880f82a
>2014-08-07 17:54:50,689 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID
>2014-08-07 17:54:50,690 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] -- copyImage parameters:
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                vmGUID=9042bb21-3f27-48bb-ab26-c0d8b18c840d
>                srcImageGUID=cc7b5e15-0557-460a-af76-7daa89c27d9f
>                srcVolUUID=1efc6019-96fd-4b59-8289-6ce7fe14a614
>                dstImageGUID=a5dac258-8f4f-48ae-b8a0-1e58a6b190db
>                dstVolUUID=46253726-8595-4414-b570-faf47fd84c4f
>                descr=Active VM
>                dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>2014-08-07 17:54:51,214 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 7880f82a
>2014-08-07 17:54:51,264 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-9) [63c518d5] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c
>2014-08-07 17:54:51,265 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-9) [63c518d5] CommandMultiAsyncTasks::AttachTask: Attaching task 0e65f6c1-4537-402b-b37f-abecd8640972 to command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c.
>2014-08-07 17:54:51,281 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-9) [63c518d5] Adding task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-08-07 17:54:51,365 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: 12896a2a, Job ID: 9aeb32de-ebca-494d-b093-f812995847a0, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blah from VM ovirtnode was initiated by admin.
>2014-08-07 17:54:51,366 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-9) BaseAsyncTask::startPollingTask: Starting to poll task 0e65f6c1-4537-402b-b37f-abecd8640972.
>2014-08-07 17:54:53,160 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
>2014-08-07 17:54:53,173 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-56) SPMAsyncTask::PollTask: Polling task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running.
>2014-08-07 17:54:53,188 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Finished polling Tasks, will poll again in 10 seconds.
>2014-08-07 17:55:34,471 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack has network interface which exceeded the defined threshold [95%] (em1: transmit rate[0%], receive rate [98%])
>2014-08-07 17:56:41,109 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-07 17:56:41,178 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-35) SPMAsyncTask::PollTask: Polling task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-08-07 17:56:41,189 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-35) BaseAsyncTask::OnTaskEndSuccess: Task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-08-07 17:56:41,192 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-35) CommandAsyncTask::EndActionIfNecessary: All tasks of command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c has ended -> executing endAction
>2014-08-07 17:56:41,194 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-35) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c): calling endAction .
>2014-08-07 17:56:41,197 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmTemplate, executionIndex: 0
>2014-08-07 17:56:41,289 INFO  [org.ovirt.engine.core.bll.AddVmTemplateCommand] (org.ovirt.thread.pool-8-thread-3) [12896a2a] Ending command successfully: org.ovirt.engine.core.bll.AddVmTemplateCommand
>2014-08-07 17:56:41,304 INFO  [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] Ending command successfully: org.ovirt.engine.core.bll.CreateImageTemplateCommand
>2014-08-07 17:56:41,316 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = a5dac258-8f4f-48ae-b8a0-1e58a6b190db, imageId = 46253726-8595-4414-b570-faf47fd84c4f), log id: 563ce49e
>2014-08-07 17:56:41,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@2455c18, log id: 563ce49e
>2014-08-07 17:56:41,358 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SetVmStatusVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, status = Down, exit status = Normal), log id: 5376710b
>2014-08-07 17:56:41,364 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SetVmStatusVDSCommand, log id: 5376710b
>2014-08-07 17:56:41,393 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) Correlation ID: 12896a2a, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blah from VM ovirtnode has been completed.
>2014-08-07 17:56:41,394 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate completed, handling the result.
>2014-08-07 17:56:41,395 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate succeeded, clearing tasks.
>2014-08-07 17:56:41,400 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-3) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 0e65f6c1-4537-402b-b37f-abecd8640972
>2014-08-07 17:56:41,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 0e65f6c1-4537-402b-b37f-abecd8640972), log id: 304f38e2
>2014-08-07 17:56:41,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=0e65f6c1-4537-402b-b37f-abecd8640972), log id: 70c2d12b
>2014-08-07 17:56:41,492 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, HSMClearTaskVDSCommand, log id: 70c2d12b
>2014-08-07 17:56:41,493 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SPMClearTaskVDSCommand, log id: 304f38e2
>2014-08-07 17:56:41,509 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-3) BaseAsyncTask::RemoveTaskFromDB: Removed task 0e65f6c1-4537-402b-b37f-abecd8640972 from DataBase
>2014-08-07 17:56:41,510 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c
>2014-08-07 17:57:52,688 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Setting new tasks map. The map contains now 0 tasks
>2014-08-07 17:57:52,690 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-08-07 18:03:52,809 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 18:03:52,816 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:03:52,865 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [39a65cfa] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:03:52,875 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [1b896c21] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:03:52,887 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [1b896c21] Correlation ID: 1b896c21, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:03:53,313 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 18:03:53,385 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2c4560d5] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:03:53,392 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [3a7e065f] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:03:53,396 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [3a7e065f] Correlation ID: 3a7e065f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:03:58,927 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 18:03:58,937 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:03:59,017 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [5ff7bf3d] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:03:59,026 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [600b0838] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:03:59,035 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [600b0838] Correlation ID: 600b0838, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:03:59,451 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 18:03:59,469 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [9e9b557] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:03:59,479 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [14f1a619] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:03:59,488 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [14f1a619] Correlation ID: 14f1a619, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:04:30,853 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 18:04:30,857 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:04:31,754 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [73365666] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:04:31,759 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [1467f976] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:04:31,762 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1467f976] Correlation ID: 1467f976, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:04:54,135 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 18:04:54,140 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:04:55,034 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [40f1557b] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:04:55,038 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [397f7bb0] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:04:55,042 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [397f7bb0] Correlation ID: 397f7bb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:04:55,470 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 18:04:55,476 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [71c19599] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:04:55,488 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [316b6c88] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:04:55,499 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [316b6c88] Correlation ID: 316b6c88, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:05:10,985 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 18:05:10,989 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:05:11,029 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [b2589ce] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:05:11,033 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [3fa8c82d] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:05:11,039 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3fa8c82d] Correlation ID: 3fa8c82d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:05:27,397 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 18:05:27,401 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:05:27,419 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [4967df9d] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:05:27,422 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [1d32d0ed] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:05:27,426 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [1d32d0ed] Correlation ID: 1d32d0ed, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:05:39,454 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-07 18:05:39,457 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:05:39,466 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-1) Operation Failed: Entity not found: foobar
>2014-08-07 18:05:39,509 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [15f82765] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:05:39,513 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [533cd946] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:05:39,517 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [533cd946] Correlation ID: 533cd946, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:05:53,363 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 18:05:53,367 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:05:53,372 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [1b92a7a6] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:05:53,375 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [48459a6b] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:05:53,379 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [48459a6b] Correlation ID: 48459a6b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:06:13,651 INFO  [org.ovirt.engine.core.bll.AddTagCommand] (ajp--127.0.0.1-8702-11) [10b44eca] Running command: AddTagCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 18:06:13,700 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [10b44eca] Correlation ID: 10b44eca, Call Stack: null, Custom Event ID: -1, Message: New Tag aaa was created by admin.
>2014-08-07 18:06:57,998 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-07 18:06:58,003 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:06:58,011 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [58d6d51] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:06:58,015 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [2acf0930] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:06:58,021 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2acf0930] Correlation ID: 2acf0930, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:07:03,741 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 18:07:03,746 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:07:03,782 INFO  [org.ovirt.engine.core.bll.AttachTemplatesToTagCommand] (ajp--127.0.0.1-8702-6) [4c1f7496] Running command: AttachTemplatesToTagCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 18:07:03,828 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [4c1f7496] Correlation ID: 4c1f7496, Call Stack: null, Custom Event ID: -1, Message: Tag aaa was attached to Templates(s) blah by admin.
>2014-08-07 18:07:04,523 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [3e1c19c8] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:07:04,525 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [31d6b45e] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:07:04,529 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [31d6b45e] Correlation ID: 31d6b45e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:07:26,772 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 18:07:26,776 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:07:27,593 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [4ac25a62] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:07:27,596 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [7a6d7981] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:07:27,601 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [7a6d7981] Correlation ID: 7a6d7981, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:07:40,932 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 18:07:40,942 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:07:41,694 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [ac1d015] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:07:41,698 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [23ddc50c] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:07:41,704 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [23ddc50c] Correlation ID: 23ddc50c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:08:17,198 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
>2014-08-07 18:08:17,202 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:09:28,422 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 18:09:28,427 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:09:28,439 INFO  [org.ovirt.engine.core.bll.AddTagCommand] (ajp--127.0.0.1-8702-11) [7d6aea68] Running command: AddTagCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 18:09:28,464 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [7d6aea68] Correlation ID: 7d6aea68, Call Stack: null, Custom Event ID: -1, Message: New Tag sracka was created by admin.
>2014-08-07 18:09:28,471 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [10f509a2] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:09:28,477 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [5e33aeb] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:09:28,484 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [5e33aeb] Correlation ID: 5e33aeb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:09:37,195 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false.
>2014-08-07 18:09:37,199 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:09:37,219 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [a08f40d] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:09:37,227 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1a6fbf5] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:09:37,240 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1a6fbf5] Correlation ID: 1a6fbf5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:09:54,712 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false.
>2014-08-07 18:09:54,717 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:09:54,733 INFO  [org.ovirt.engine.core.bll.AttachTemplatesToTagCommand] (ajp--127.0.0.1-8702-3) [7d689210] Running command: AttachTemplatesToTagCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 18:09:54,747 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7d689210] Correlation ID: 7d689210, Call Stack: null, Custom Event ID: -1, Message: Tag sracka was attached to Templates(s) blah by admin.
>2014-08-07 18:09:54,754 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [12dc978e] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:09:54,757 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [10d9cc95] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:09:54,762 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [10d9cc95] Correlation ID: 10d9cc95, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:10:02,610 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
>2014-08-07 18:10:02,613 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:10:02,632 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2027d0e9] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:10:02,641 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [4389c5d3] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:10:02,644 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [4389c5d3] Correlation ID: 4389c5d3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:10:17,237 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 18:10:17,248 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:10:17,627 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-07 18:10:17,633 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6a933c5d] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:10:17,638 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [70bfc990] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:10:17,644 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [70bfc990] Correlation ID: 70bfc990, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:10:36,145 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
>2014-08-07 18:10:36,149 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:10:36,159 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [453ac2dd] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:10:36,165 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [13dc328] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:10:36,170 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [13dc328] Correlation ID: 13dc328, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:10:43,121 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
>2014-08-07 18:10:43,125 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:10:43,151 INFO  [org.ovirt.engine.core.bll.DetachTemplateFromTagCommand] (ajp--127.0.0.1-8702-10) [604241ea] Running command: DetachTemplateFromTagCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-07 18:10:43,174 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [604241ea] Correlation ID: 604241ea, Call Stack: null, Custom Event ID: -1, Message: Tag aaa was detached from Template(s) blah by admin.
>2014-08-07 18:10:43,179 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [4a4be5e] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:10:43,184 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [38a6afda] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:10:43,189 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [38a6afda] Correlation ID: 38a6afda, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:11:01,080 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false.
>2014-08-07 18:11:01,093 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:11:01,111 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7c793aad] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:11:01,115 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [7e118e5c] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:11:01,121 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [7e118e5c] Correlation ID: 7e118e5c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:11:04,478 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false.
>2014-08-07 18:11:04,502 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [7b79c73] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:11:04,505 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1c4fbc26] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:11:04,509 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1c4fbc26] Correlation ID: 1c4fbc26, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:11:18,531 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false.
>2014-08-07 18:11:18,537 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-07 18:11:18,544 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [60fc0d21] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:11:18,547 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [30286889] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:11:18,551 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [30286889] Correlation ID: 30286889, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:11:18,886 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
>2014-08-07 18:11:18,892 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [39a55340] Running command: LogoutBySessionCommand internal: false.
>2014-08-07 18:11:18,895 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [7aa1feb0] Running command: LogoutUserCommand internal: false.
>2014-08-07 18:11:18,898 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [7aa1feb0] Correlation ID: 7aa1feb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-07 18:20:41,078 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-07 18:23:48,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
>2014-08-07 18:23:48,980 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued.
>2014-08-07 18:23:48,983 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
>	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.net.ConnectException: Connection refused
>	at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55]
>	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55]
>	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55]
>	at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55]
>	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:]
>	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
>	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
>	at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
>	at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
>	at com.sun.proxy.$Proxy125.list(Unknown Source)
>	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
>	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
>
>2014-08-07 18:23:49,024 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing.
>java.net.ConnectException: Connection refused
>2014-08-07 18:23:52,403 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : no reporting hosts
>2014-08-07 18:23:52,407 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts
>2014-08-07 18:23:52,411 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 63fe3516-4170-402a-9dc8-6d3eab1c7933 status from Active to Unknown, reason : no reporting hosts
>2014-08-07 18:23:52,497 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 18:23:52,522 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 23690360
>2014-08-07 18:23:52,525 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [7f085c60] Correlation ID: 7f085c60, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-07 18:23:52,544 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 
> 
>TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
>
>2014-08-07 18:23:52,548 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] HostName = rack
>2014-08-07 18:23:52,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
>2014-08-07 18:23:52,694 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, GetHardwareInfoVDSCommand, log id: 23690360
>2014-08-07 18:23:52,744 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [aad989f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 18:23:52,763 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [22642369] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 18:23:52,791 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-07 18:23:52,817 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c970424
>2014-08-07 18:23:52,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6c970424
>2014-08-07 18:23:52,851 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Host rack storage connection was succeeded 
>2014-08-07 18:23:52,868 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 61f06e9
>2014-08-07 18:23:53,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 61f06e9
>2014-08-07 18:23:53,369 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d18d97b
>2014-08-07 18:23:53,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1d18d97b
>2014-08-07 18:23:53,469 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-07 18:23:53,476 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-07 18:23:53,501 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [215ad4c4] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-07 18:24:02,750 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-07 18:24:02,766 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-07 18:24:02,771 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42cf5663
>2014-08-07 18:24:02,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling started: taskId = 4d485da8-e4b0-4ec2-97f9-cf7eac7579a4
>2014-08-07 18:24:03,829 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling ended: taskId = 4d485da8-e4b0-4ec2-97f9-cf7eac7579a4 task status = finished
>2014-08-07 18:24:03,885 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling ended, spm status: SPM
>2014-08-07 18:24:03,888 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-99) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4d485da8-e4b0-4ec2-97f9-cf7eac7579a4), log id: 11235c5d
>2014-08-07 18:24:03,934 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, HSMClearTaskVDSCommand, log id: 11235c5d
>2014-08-07 18:24:03,935 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65e9d366, log id: 42cf5663
>2014-08-07 18:24:03,938 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-07 18:24:03,944 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-07 18:24:03,958 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 17655a77
>2014-08-07 18:24:04,004 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9
>2014-08-07 18:24:04,007 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 14e821e3
>2014-08-07 18:24:04,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14e821e3
>2014-08-07 18:24:04,013 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17655a77
>2014-08-07 18:24:04,014 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-19) Discovered no tasks on Storage Pool Default
>2014-08-07 18:37:52,684 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized
>2014-08-07 18:37:53,681 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf.
>2014-08-07 18:37:53,815 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-08-07 18:37:53,820 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 18:37:53,866 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 2dae23a2
>2014-08-07 18:37:53,868 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-08-07 18:37:53,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014"}
>
>2014-08-07 18:37:54,002 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] FINISH, SetVolumeDescriptionVDSCommand, log id: 2dae23a2
>2014-08-07 18:37:54,076 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 18:37:54,085 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5e622a41
>2014-08-07 18:37:54,086 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeVdsBrokerCommand: 
>2014-08-07 18:37:54,088 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                size=20,480
>
>2014-08-07 18:37:54,710 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, UploadStreamVDSCommand, log id: 5e622a41
>2014-08-07 18:37:54,732 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 29253b22-c73a-4328-a4e2-43d6756e8cfa
>2014-08-07 18:37:54,733 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] CommandMultiAsyncTasks::AttachTask: Attaching task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 to command 29253b22-c73a-4328-a4e2-43d6756e8cfa.
>2014-08-07 18:37:54,743 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Adding task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-08-07 18:37:54,750 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-08-07 18:37:54,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 134d15b3
>2014-08-07 18:37:54,767 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-08-07 18:37:54,768 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717
>                volUUID=8596469e-37df-4d9e-9793-d238238131f5
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Thu Aug 07 18:37:53 CEST 2014","Size":20480}
>
>2014-08-07 18:37:54,902 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, SetVolumeDescriptionVDSCommand, log id: 134d15b3
>2014-08-07 18:37:54,914 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 36132e09
>2014-08-07 18:37:54,923 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-08-07 18:37:54,929 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014"}
>
>2014-08-07 18:37:55,073 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, SetVolumeDescriptionVDSCommand, log id: 36132e09
>2014-08-07 18:37:55,099 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Running command: UploadStreamCommand internal: true. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-07 18:37:55,103 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: b6c7c5d
>2014-08-07 18:37:55,106 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- executeVdsBrokerCommand: 
>2014-08-07 18:37:55,107 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                size=20,480
>
>2014-08-07 18:37:55,763 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] FINISH, UploadStreamVDSCommand, log id: b6c7c5d
>2014-08-07 18:37:55,766 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] CommandMultiAsyncTasks::AttachTask: Attaching task 982f8689-dddf-4b8a-bd50-d1f2c79be801 to command 29253b22-c73a-4328-a4e2-43d6756e8cfa.
>2014-08-07 18:37:55,799 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Adding task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet..
>2014-08-07 18:37:55,808 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-08-07 18:37:55,809 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-08-07 18:37:55,811 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-08-07 18:37:55,812 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-08-07 18:37:55,820 INFO  [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-08-07 18:37:55,830 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 6ca80435
>2014-08-07 18:37:55,833 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- executeIrsBrokerCommand: calling 'setVolumeDescription' 
>2014-08-07 18:37:55,835 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- setVolumeDescription parameters:
>                spUUID=00000002-0002-0002-0002-0000000001d9
>                sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752
>                imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216
>                volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21
>                description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Thu Aug 07 18:37:53 CEST 2014","Size":20480}
>
>2014-08-07 18:37:55,968 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] FINISH, SetVolumeDescriptionVDSCommand, log id: 6ca80435
>2014-08-07 18:37:55,972 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] BaseAsyncTask::startPollingTask: Starting to poll task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9.
>2014-08-07 18:37:55,973 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] BaseAsyncTask::startPollingTask: Starting to poll task 982f8689-dddf-4b8a-bd50-d1f2c79be801.
>2014-08-07 18:38:01,290 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-55) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now
>2014-08-07 18:38:01,296 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) SPMAsyncTask::PollTask: Polling task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-08-07 18:38:01,299 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) BaseAsyncTask::OnTaskEndSuccess: Task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-08-07 18:38:01,301 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-55) Task with DB Task ID 2134bb45-6102-4fe9-9913-ddb0137cb290 and VDSM Task ID 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 is in state Polling. End action for command 29253b22-c73a-4328-a4e2-43d6756e8cfa will proceed when all the entitys tasks are completed.
>2014-08-07 18:38:01,303 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) SPMAsyncTask::PollTask: Polling task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'.
>2014-08-07 18:38:01,304 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) BaseAsyncTask::OnTaskEndSuccess: Task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully.
>2014-08-07 18:38:01,305 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-55) CommandAsyncTask::EndActionIfNecessary: All tasks of command 29253b22-c73a-4328-a4e2-43d6756e8cfa has ended -> executing endAction
>2014-08-07 18:38:01,306 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-55) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 29253b22-c73a-4328-a4e2-43d6756e8cfa): calling endAction .
>2014-08-07 18:38:01,308 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0
>2014-08-07 18:38:01,345 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand
>2014-08-07 18:38:01,346 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist
>2014-08-07 18:38:01,348 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK
>2014-08-07 18:38:01,349 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE
>2014-08-07 18:38:01,350 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK
>2014-08-07 18:38:01,352 INFO  [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK
>key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK
>, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE
>]
>2014-08-07 18:38:01,353 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result.
>2014-08-07 18:38:01,355 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks.
>2014-08-07 18:38:01,356 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 982f8689-dddf-4b8a-bd50-d1f2c79be801
>2014-08-07 18:38:01,357 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 982f8689-dddf-4b8a-bd50-d1f2c79be801), log id: 7bc1f07e
>2014-08-07 18:38:01,363 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=982f8689-dddf-4b8a-bd50-d1f2c79be801), log id: 6bab51e8
>2014-08-07 18:38:01,414 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, HSMClearTaskVDSCommand, log id: 6bab51e8
>2014-08-07 18:38:01,415 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, SPMClearTaskVDSCommand, log id: 7bc1f07e
>2014-08-07 18:38:01,422 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] BaseAsyncTask::RemoveTaskFromDB: Removed task 982f8689-dddf-4b8a-bd50-d1f2c79be801 from DataBase
>2014-08-07 18:38:01,423 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9
>2014-08-07 18:38:01,424 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9), log id: 183344c8
>2014-08-07 18:38:01,427 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9), log id: 4de8b2e1
>2014-08-07 18:38:01,496 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, HSMClearTaskVDSCommand, log id: 4de8b2e1
>2014-08-07 18:38:01,497 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, SPMClearTaskVDSCommand, log id: 183344c8
>2014-08-07 18:38:01,502 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] BaseAsyncTask::RemoveTaskFromDB: Removed task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 from DataBase
>2014-08-07 18:38:01,503 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 29253b22-c73a-4328-a4e2-43d6756e8cfa
>2014-08-07 18:39:22,725 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Setting new tasks map. The map contains now 0 tasks
>2014-08-07 18:39:22,726 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9.
>2014-08-07 18:42:52,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21ab8d34
>2014-08-07 18:42:52,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 21ab8d34
>2014-08-07 18:42:52,187 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 19:37:52,699 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-98) Principal admin::internal synchronized
>2014-08-07 19:37:55,974 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf.
>2014-08-07 19:42:52,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e262ca
>2014-08-07 19:42:52,223 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2e262ca
>2014-08-07 19:42:52,228 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-23) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 20:37:52,705 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized
>2014-08-07 20:37:55,995 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
>2014-08-07 20:42:52,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [46247d52] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 36d349dd
>2014-08-07 20:42:52,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [46247d52] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 36d349dd
>2014-08-07 20:42:52,259 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) [46247d52] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 21:37:52,712 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7f085c60] Principal admin::internal synchronized
>2014-08-07 21:37:56,033 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf.
>2014-08-07 21:42:52,273 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e77b3f
>2014-08-07 21:42:52,294 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2e77b3f
>2014-08-07 21:42:52,301 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 22:37:52,720 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized
>2014-08-07 22:37:56,073 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) [46247d52] Attempting to update VMs/Templates Ovf.
>2014-08-07 22:42:52,312 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 51ddef45
>2014-08-07 22:42:52,329 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 51ddef45
>2014-08-07 22:42:52,336 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-07 23:37:52,727 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized
>2014-08-07 23:37:56,116 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
>2014-08-07 23:42:52,344 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41f6b132
>2014-08-07 23:42:52,360 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 41f6b132
>2014-08-07 23:42:52,366 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 00:37:52,733 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized
>2014-08-08 00:37:56,138 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf.
>2014-08-08 00:42:52,375 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28c33dd6
>2014-08-08 00:42:52,393 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28c33dd6
>2014-08-08 00:42:52,398 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 01:37:52,867 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized
>2014-08-08 01:37:56,156 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-08-08 01:42:52,406 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f6c7da7
>2014-08-08 01:42:52,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: f6c7da7
>2014-08-08 01:42:52,427 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-15) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 02:37:52,880 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-08 02:37:56,170 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
>2014-08-08 02:42:52,437 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 732fc359
>2014-08-08 02:42:52,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 732fc359
>2014-08-08 02:42:52,459 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) [175991ea] Start deleteAgedOutAuditLogs
>2014-08-08 03:35:35,004 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) Start deleteAgedOutCommandEntities
>2014-08-08 03:35:35,024 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) Finished deleteAgedOutCommandEntities
>2014-08-08 03:35:35,026 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) [175991ea] Finished deleteAgedOutAuditLogs
>2014-08-08 03:37:52,886 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-93) Principal admin::internal synchronized
>2014-08-08 03:37:56,194 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
>2014-08-08 03:42:52,469 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2f159380
>2014-08-08 03:42:52,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2f159380
>2014-08-08 03:42:52,492 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-41) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 04:37:52,894 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized
>2014-08-08 04:37:56,257 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf.
>2014-08-08 04:42:52,500 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7996de58
>2014-08-08 04:42:52,517 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7996de58
>2014-08-08 04:42:52,522 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 05:37:52,907 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7f085c60] Principal admin::internal synchronized
>2014-08-08 05:37:56,306 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf.
>2014-08-08 05:42:52,533 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6397d16
>2014-08-08 05:42:52,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6397d16
>2014-08-08 05:42:52,556 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 06:37:52,913 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized
>2014-08-08 06:37:56,326 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf.
>2014-08-08 06:42:52,564 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 64eaa2a3
>2014-08-08 06:42:52,582 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 64eaa2a3
>2014-08-08 06:42:52,587 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-8) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 07:37:52,918 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized
>2014-08-08 07:37:56,348 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf.
>2014-08-08 07:42:52,595 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7d7a0681
>2014-08-08 07:42:52,612 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7d7a0681
>2014-08-08 07:42:52,617 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 08:37:52,924 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized
>2014-08-08 08:37:56,385 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf.
>2014-08-08 08:42:52,621 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 38ec632d
>2014-08-08 08:42:52,639 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 38ec632d
>2014-08-08 08:42:52,644 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-30) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 09:37:52,941 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized
>2014-08-08 09:37:56,424 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-08-08 09:42:52,655 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 508fa176
>2014-08-08 09:42:52,671 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 508fa176
>2014-08-08 09:42:52,677 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 10:37:52,947 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized
>2014-08-08 10:37:56,445 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf.
>2014-08-08 10:42:52,686 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4b7e124e
>2014-08-08 10:42:52,702 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4b7e124e
>2014-08-08 10:42:52,708 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-5) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 11:34:25,181 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginAdminUserCommand internal: false.
>2014-08-08 11:34:25,190 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-08 11:34:38,405 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
>2014-08-08 11:34:38,416 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-08 11:37:52,953 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) [215ad4c4] Principal admin::internal synchronized
>2014-08-08 11:37:56,463 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-08-08 11:42:52,721 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 79c94bd6
>2014-08-08 11:42:52,737 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 79c94bd6
>2014-08-08 11:42:52,748 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 12:37:52,960 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized
>2014-08-08 12:37:56,495 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf.
>2014-08-08 12:42:52,757 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10adc186
>2014-08-08 12:42:52,778 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 10adc186
>2014-08-08 12:42:52,784 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-10) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 13:27:07,400 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-8) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-08 13:37:52,966 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized
>2014-08-08 13:37:56,517 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf.
>2014-08-08 13:42:52,793 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 281ff8c3
>2014-08-08 13:42:52,807 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 281ff8c3
>2014-08-08 13:42:52,813 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-10) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 14:37:52,974 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized
>2014-08-08 14:37:56,537 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
>2014-08-08 14:42:52,821 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 26a95266
>2014-08-08 14:42:52,838 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 26a95266
>2014-08-08 14:42:52,842 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 15:12:46,620 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [170cbf09] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-08 15:12:46,684 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] Running command: DeactivateStorageDomainCommand internal: false. Entities affected :  ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage
>2014-08-08 15:12:46,695 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 6a2185c6
>2014-08-08 15:12:46,708 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] FINISH, DeactivateStorageDomainVDSCommand, log id: 6a2185c6
>2014-08-08 15:12:46,709 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-08 15:12:46,723 INFO  [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-9) After deactivate treatment vds: rack,pool Default
>2014-08-08 15:12:46,725 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: a66fe89
>2014-08-08 15:12:47,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, ConnectStoragePoolVDSCommand, log id: a66fe89
>2014-08-08 15:12:47,066 INFO  [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-08 15:12:47,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7add075a
>2014-08-08 15:12:47,314 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7add075a
>2014-08-08 15:12:47,316 INFO  [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] Domain 44559ec0-6bef-491c-bbd6-e67278f37987 will remain in PreparingForMaintenance status until deactivated on all hosts
>2014-08-08 15:12:47,334 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [3732573e] Correlation ID: 170cbf09, Job ID: c7733ca5-9c01-46b5-af1f-e5db0f5a8720, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin
>2014-08-08 15:12:54,819 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache
>2014-08-08 15:13:02,340 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance
>2014-08-08 15:37:52,980 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized
>2014-08-08 15:37:56,558 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf.
>2014-08-08 15:42:52,851 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4fe40d4c
>2014-08-08 15:42:52,867 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4fe40d4c
>2014-08-08 15:42:52,872 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 16:37:52,986 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized
>2014-08-08 16:37:56,602 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf.
>2014-08-08 16:42:52,881 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7ab331b0
>2014-08-08 16:42:52,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7ab331b0
>2014-08-08 16:42:52,902 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-29) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 17:37:52,995 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized
>2014-08-08 17:37:56,628 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
>2014-08-08 17:42:52,913 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-43) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 768ea522
>2014-08-08 17:42:52,931 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-43) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 768ea522
>2014-08-08 17:42:52,937 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-43) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 18:37:53,004 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized
>2014-08-08 18:37:56,661 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf.
>2014-08-08 18:39:56,867 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-08 18:42:52,948 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 608b195c
>2014-08-08 18:42:52,963 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 608b195c
>2014-08-08 18:42:52,969 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-13) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 19:37:53,014 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized
>2014-08-08 19:37:56,685 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-08-08 19:42:52,978 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 555199ad
>2014-08-08 19:42:52,995 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 555199ad
>2014-08-08 19:42:53,001 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 20:37:53,021 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized
>2014-08-08 20:37:56,720 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
>2014-08-08 20:42:53,012 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 12b46b69
>2014-08-08 20:42:53,028 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 12b46b69
>2014-08-08 20:42:53,034 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-5) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 21:37:53,026 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-08 21:37:56,740 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf.
>2014-08-08 21:42:53,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6b006980
>2014-08-08 21:42:53,062 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6b006980
>2014-08-08 21:42:53,068 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 22:37:53,032 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized
>2014-08-08 22:37:56,779 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-08-08 22:42:53,077 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d66e654
>2014-08-08 22:42:53,093 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1d66e654
>2014-08-08 22:42:53,098 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-08 23:37:53,035 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized
>2014-08-08 23:37:56,803 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf.
>2014-08-08 23:42:53,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 717c9cb4
>2014-08-08 23:42:53,125 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 717c9cb4
>2014-08-08 23:42:53,130 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-44) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 00:37:53,043 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized
>2014-08-09 00:37:56,823 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-09 00:42:53,140 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 47cc8e8c
>2014-08-09 00:42:53,158 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 47cc8e8c
>2014-08-09 00:42:53,165 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 01:37:53,051 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized
>2014-08-09 01:37:56,857 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf.
>2014-08-09 01:42:53,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: fab49ec
>2014-08-09 01:42:53,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: fab49ec
>2014-08-09 01:42:53,199 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 02:37:53,059 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized
>2014-08-09 02:37:56,878 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-08-09 02:42:53,212 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4f761905
>2014-08-09 02:42:53,228 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4f761905
>2014-08-09 02:42:53,235 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 03:35:35,003 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutAuditLogs
>2014-08-09 03:35:35,003 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-70) Start deleteAgedOutCommandEntities
>2014-08-09 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-70) Finished deleteAgedOutCommandEntities
>2014-08-09 03:35:35,011 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutAuditLogs
>2014-08-09 03:37:53,064 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized
>2014-08-09 03:37:56,896 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf.
>2014-08-09 03:42:53,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6c97c0ab
>2014-08-09 03:42:53,260 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6c97c0ab
>2014-08-09 03:42:53,266 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 04:37:53,073 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-09 04:37:56,918 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-08-09 04:42:53,277 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7cd1d4f4
>2014-08-09 04:42:53,299 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7cd1d4f4
>2014-08-09 04:42:53,308 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-16) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 05:37:53,079 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized
>2014-08-09 05:37:56,950 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf.
>2014-08-09 05:42:53,319 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 30eb1cc7
>2014-08-09 05:42:53,336 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 30eb1cc7
>2014-08-09 05:42:53,341 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 06:37:53,089 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized
>2014-08-09 06:37:56,970 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
>2014-08-09 06:42:53,354 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4df0ce86
>2014-08-09 06:42:53,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4df0ce86
>2014-08-09 06:42:53,379 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 07:37:53,094 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized
>2014-08-09 07:37:57,002 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
>2014-08-09 07:42:53,390 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 70fe66e2
>2014-08-09 07:42:53,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 70fe66e2
>2014-08-09 07:42:53,414 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-37) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 08:37:53,105 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized
>2014-08-09 08:37:57,023 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf.
>2014-08-09 08:42:53,421 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: bb1a5d9
>2014-08-09 08:42:53,439 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: bb1a5d9
>2014-08-09 08:42:53,444 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-28) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 08:46:35,420 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-09 09:37:53,113 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized
>2014-08-09 09:37:57,041 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf.
>2014-08-09 09:42:53,454 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6943cb34
>2014-08-09 09:42:53,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6943cb34
>2014-08-09 09:42:53,476 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-33) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 10:37:53,121 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized
>2014-08-09 10:37:57,078 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf.
>2014-08-09 10:42:53,485 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1c0be4c5
>2014-08-09 10:42:53,503 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1c0be4c5
>2014-08-09 10:42:53,510 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 11:37:53,131 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized
>2014-08-09 11:37:57,099 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
>2014-08-09 11:42:53,521 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5746ee64
>2014-08-09 11:42:53,539 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5746ee64
>2014-08-09 11:42:53,546 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-1) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 12:37:53,139 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized
>2014-08-09 12:37:57,131 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
>2014-08-09 12:42:53,563 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 44a7aa9b
>2014-08-09 12:42:53,579 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 44a7aa9b
>2014-08-09 12:42:53,585 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 13:37:53,146 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized
>2014-08-09 13:37:57,153 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf.
>2014-08-09 13:42:53,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5dcdd181
>2014-08-09 13:42:53,614 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5dcdd181
>2014-08-09 13:42:53,620 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-28) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 14:37:53,154 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized
>2014-08-09 14:37:57,175 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-08-09 14:42:53,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6e87be8b
>2014-08-09 14:42:53,645 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6e87be8b
>2014-08-09 14:42:53,650 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 15:37:53,161 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized
>2014-08-09 15:37:57,210 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-09 15:42:53,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28bd6d77
>2014-08-09 15:42:53,674 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28bd6d77
>2014-08-09 15:42:53,679 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 16:37:53,166 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized
>2014-08-09 16:37:57,233 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
>2014-08-09 16:42:53,691 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7d65f981
>2014-08-09 16:42:53,707 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7d65f981
>2014-08-09 16:42:53,713 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 17:37:53,172 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-09 17:37:57,269 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
>2014-08-09 17:42:53,722 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5298f950
>2014-08-09 17:42:53,738 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5298f950
>2014-08-09 17:42:53,743 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 18:37:53,177 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized
>2014-08-09 18:37:57,289 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
>2014-08-09 18:42:53,754 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 712a41c7
>2014-08-09 18:42:53,772 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 712a41c7
>2014-08-09 18:42:53,780 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 19:37:53,182 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized
>2014-08-09 19:37:57,309 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf.
>2014-08-09 19:42:53,788 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 68e2d83c
>2014-08-09 19:42:53,804 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 68e2d83c
>2014-08-09 19:42:53,811 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 20:37:53,187 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized
>2014-08-09 20:37:57,328 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
>2014-08-09 20:42:53,820 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f516bee
>2014-08-09 20:42:53,836 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f516bee
>2014-08-09 20:42:53,841 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 21:37:53,192 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized
>2014-08-09 21:37:57,363 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf.
>2014-08-09 21:42:53,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2020d46f
>2014-08-09 21:42:53,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2020d46f
>2014-08-09 21:42:53,875 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-25) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 22:37:53,198 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized
>2014-08-09 22:37:57,384 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-08-09 22:42:53,884 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 51be8099
>2014-08-09 22:42:53,899 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 51be8099
>2014-08-09 22:42:53,906 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-14) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-09 23:37:53,206 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized
>2014-08-09 23:37:57,405 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf.
>2014-08-09 23:42:53,915 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4ae3e649
>2014-08-09 23:42:53,933 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4ae3e649
>2014-08-09 23:42:53,941 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 00:37:53,214 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized
>2014-08-10 00:37:57,423 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf.
>2014-08-10 00:42:53,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-49) [7f085c60] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 504970c3
>2014-08-10 00:42:53,967 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-49) [7f085c60] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 504970c3
>2014-08-10 00:42:53,973 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-49) [7f085c60] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 01:37:53,221 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized
>2014-08-10 01:37:57,462 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf.
>2014-08-10 01:42:53,986 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7956fbf1
>2014-08-10 01:42:54,003 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7956fbf1
>2014-08-10 01:42:54,009 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-25) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 02:37:53,227 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized
>2014-08-10 02:37:57,484 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-08-10 02:42:54,017 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 67158e84
>2014-08-10 02:42:54,033 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 67158e84
>2014-08-10 02:42:54,037 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-3) Start deleteAgedOutAuditLogs
>2014-08-10 03:35:35,003 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) Start deleteAgedOutCommandEntities
>2014-08-10 03:35:35,011 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-3) Finished deleteAgedOutAuditLogs
>2014-08-10 03:35:35,015 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) Finished deleteAgedOutCommandEntities
>2014-08-10 03:37:53,236 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized
>2014-08-10 03:37:57,512 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
>2014-08-10 03:42:54,044 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 62baab2b
>2014-08-10 03:42:54,061 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 62baab2b
>2014-08-10 03:42:54,066 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 04:37:53,244 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized
>2014-08-10 04:37:57,531 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf.
>2014-08-10 04:42:54,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27e3011b
>2014-08-10 04:42:54,091 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 27e3011b
>2014-08-10 04:42:54,096 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-20) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 05:37:53,252 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized
>2014-08-10 05:37:57,554 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf.
>2014-08-10 05:42:54,104 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 560db072
>2014-08-10 05:42:54,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 560db072
>2014-08-10 05:42:54,126 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 06:37:53,261 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized
>2014-08-10 06:37:57,591 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf.
>2014-08-10 06:42:54,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 53cd5519
>2014-08-10 06:42:54,149 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 53cd5519
>2014-08-10 06:42:54,153 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 07:37:53,270 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized
>2014-08-10 07:37:57,618 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-10 07:42:54,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f28cd5d
>2014-08-10 07:42:54,181 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f28cd5d
>2014-08-10 07:42:54,188 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 08:37:53,279 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized
>2014-08-10 08:37:57,640 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf.
>2014-08-10 08:42:54,200 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7781ebf0
>2014-08-10 08:42:54,216 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7781ebf0
>2014-08-10 08:42:54,222 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 09:37:53,288 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized
>2014-08-10 09:37:57,673 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-08-10 09:42:54,233 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1006ed2a
>2014-08-10 09:42:54,320 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1006ed2a
>2014-08-10 09:42:54,325 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-34) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 10:37:53,294 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized
>2014-08-10 10:37:57,710 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
>2014-08-10 10:42:54,338 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7206f594
>2014-08-10 10:42:54,355 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7206f594
>2014-08-10 10:42:54,362 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 11:37:53,300 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized
>2014-08-10 11:37:57,747 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf.
>2014-08-10 11:42:54,372 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f3e0d38
>2014-08-10 11:42:54,389 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f3e0d38
>2014-08-10 11:42:54,393 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 12:37:53,310 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized
>2014-08-10 12:37:57,771 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
>2014-08-10 12:42:54,404 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 492a8558
>2014-08-10 12:42:55,310 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 492a8558
>2014-08-10 12:42:55,315 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 13:37:53,316 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized
>2014-08-10 13:37:57,791 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
>2014-08-10 13:42:55,327 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1f1f2d28
>2014-08-10 13:42:55,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1f1f2d28
>2014-08-10 13:42:55,411 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-41) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 14:37:53,326 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized
>2014-08-10 14:37:57,810 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
>2014-08-10 14:42:55,422 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f601290
>2014-08-10 14:42:55,441 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: f601290
>2014-08-10 14:42:55,447 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-38) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 15:37:53,337 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized
>2014-08-10 15:37:57,849 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf.
>2014-08-10 15:42:55,461 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 580ce1
>2014-08-10 15:42:55,478 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 580ce1
>2014-08-10 15:42:55,484 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 16:37:53,343 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized
>2014-08-10 16:37:57,889 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf.
>2014-08-10 16:42:55,490 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6af9a289
>2014-08-10 16:42:55,507 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6af9a289
>2014-08-10 16:42:55,512 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 17:37:53,349 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized
>2014-08-10 17:37:57,908 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
>2014-08-10 17:42:55,519 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 377fc645
>2014-08-10 17:42:55,536 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 377fc645
>2014-08-10 17:42:55,541 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 18:37:53,358 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized
>2014-08-10 18:37:57,927 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf.
>2014-08-10 18:42:55,554 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4a563585
>2014-08-10 18:42:55,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4a563585
>2014-08-10 18:42:55,580 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-29) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 19:37:53,366 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized
>2014-08-10 19:37:57,960 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf.
>2014-08-10 19:42:55,593 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5371f179
>2014-08-10 19:42:55,610 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5371f179
>2014-08-10 19:42:55,616 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 20:37:53,375 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized
>2014-08-10 20:37:57,999 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Attempting to update VMs/Templates Ovf.
>2014-08-10 20:42:55,628 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 16eef8f3
>2014-08-10 20:42:55,644 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 16eef8f3
>2014-08-10 20:42:55,651 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-4) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 21:37:53,384 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized
>2014-08-10 21:37:58,034 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf.
>2014-08-10 21:42:55,664 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65fe5038
>2014-08-10 21:42:55,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 65fe5038
>2014-08-10 21:42:55,686 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 22:37:53,391 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized
>2014-08-10 22:37:58,073 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf.
>2014-08-10 22:42:55,695 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32dbfb15
>2014-08-10 22:42:55,758 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 32dbfb15
>2014-08-10 22:42:55,766 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-10 23:37:53,398 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized
>2014-08-10 23:37:58,091 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf.
>2014-08-10 23:42:55,779 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 707a5776
>2014-08-10 23:42:55,796 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 707a5776
>2014-08-10 23:42:55,803 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-20) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 00:37:53,404 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized
>2014-08-11 00:37:58,112 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf.
>2014-08-11 00:42:55,812 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1bed9648
>2014-08-11 00:42:55,828 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1bed9648
>2014-08-11 00:42:55,833 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 01:37:53,414 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized
>2014-08-11 01:37:58,137 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
>2014-08-11 01:42:55,843 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4f9e4a7
>2014-08-11 01:42:55,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4f9e4a7
>2014-08-11 01:42:55,864 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 02:37:53,420 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Principal admin::internal synchronized
>2014-08-11 02:37:58,172 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf.
>2014-08-11 02:42:55,877 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7bc89122
>2014-08-11 02:42:55,894 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7bc89122
>2014-08-11 02:42:55,900 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 03:35:35,001 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7f085c60] Start deleteAgedOutAuditLogs
>2014-08-11 03:35:35,002 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutCommandEntities
>2014-08-11 03:35:35,010 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7f085c60] Finished deleteAgedOutAuditLogs
>2014-08-11 03:35:35,011 INFO  [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutCommandEntities
>2014-08-11 03:37:53,428 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized
>2014-08-11 03:37:58,195 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf.
>2014-08-11 03:42:55,907 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2f02881d
>2014-08-11 03:42:55,923 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2f02881d
>2014-08-11 03:42:55,929 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-16) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 04:37:53,437 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized
>2014-08-11 04:37:58,215 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
>2014-08-11 04:42:55,938 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 12beff5c
>2014-08-11 04:42:55,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 12beff5c
>2014-08-11 04:42:55,960 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 05:37:53,444 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized
>2014-08-11 05:37:58,250 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf.
>2014-08-11 05:42:55,973 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41893219
>2014-08-11 05:42:55,990 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 41893219
>2014-08-11 05:42:55,996 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 06:37:53,450 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized
>2014-08-11 06:37:58,287 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
>2014-08-11 06:42:56,005 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 429a88fa
>2014-08-11 06:42:56,020 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 429a88fa
>2014-08-11 06:42:56,026 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-24) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 07:37:53,455 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-11 07:37:58,309 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
>2014-08-11 07:42:56,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3f3981e
>2014-08-11 07:42:56,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3f3981e
>2014-08-11 07:42:56,055 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-44) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 08:37:53,461 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized
>2014-08-11 08:37:58,328 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf.
>2014-08-11 08:42:56,065 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39df1572
>2014-08-11 08:42:56,082 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 39df1572
>2014-08-11 08:42:56,088 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 08:50:39,064 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false.
>2014-08-11 08:50:39,072 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-11 08:50:48,689 INFO  [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginAdminUserCommand internal: false.
>2014-08-11 08:50:48,694 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-11 09:02:44,116 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-2) Retrieving non string value from session
>2014-08-11 09:36:26,701 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-11 09:37:53,470 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized
>2014-08-11 09:37:58,366 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf.
>2014-08-11 09:42:56,097 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1a385435
>2014-08-11 09:42:56,121 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1a385435
>2014-08-11 09:42:56,130 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 10:37:53,478 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized
>2014-08-11 10:37:58,406 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf.
>2014-08-11 10:42:56,142 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 352e7fdd
>2014-08-11 10:42:56,159 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 352e7fdd
>2014-08-11 10:42:56,165 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 11:37:53,488 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-56) Principal admin::internal synchronized
>2014-08-11 11:37:58,441 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf.
>2014-08-11 11:42:56,174 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28eaa2b6
>2014-08-11 11:42:56,192 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28eaa2b6
>2014-08-11 11:42:56,197 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-9) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 12:37:53,495 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-89) Principal admin::internal synchronized
>2014-08-11 12:37:58,486 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf.
>2014-08-11 12:42:56,204 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3ea5c47a
>2014-08-11 12:42:56,220 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3ea5c47a
>2014-08-11 12:42:56,225 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 13:05:30,244 INFO  [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
>2014-08-11 13:05:30,256 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in.
>2014-08-11 13:05:30,266 INFO  [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [13e65a3e] Running command: LogoutBySessionCommand internal: false.
>2014-08-11 13:05:30,270 INFO  [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [489031bf] Running command: LogoutUserCommand internal: false.
>2014-08-11 13:05:30,279 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [489031bf] Correlation ID: 489031bf, Call Stack: null, Custom Event ID: -1, Message: User admin logged out.
>2014-08-11 13:37:53,503 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized
>2014-08-11 13:37:58,505 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
>2014-08-11 13:42:56,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [175991ea] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4ba7ee0d
>2014-08-11 13:42:56,254 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [175991ea] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4ba7ee0d
>2014-08-11 13:42:56,260 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-1) [175991ea] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 14:04:41,968 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-8) Retrieving non string value from session
>2014-08-11 14:37:53,513 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized
>2014-08-11 14:37:58,525 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
>2014-08-11 14:42:56,272 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4aa2e265
>2014-08-11 14:42:56,289 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4aa2e265
>2014-08-11 14:42:56,296 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-34) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 15:37:53,519 INFO  [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized
>2014-08-11 15:37:58,548 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf.
>2014-08-11 15:42:56,308 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 19730428
>2014-08-11 15:42:56,324 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 19730428
>2014-08-11 15:42:56,329 INFO  [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-30) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933.
>2014-08-11 15:57:22,598 WARN  [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue.
>2014-08-11 16:00:56,593 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-4) Retrieving non string value from session
>2014-08-11 16:01:07,107 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-8) Retrieving non string value from session
>2014-08-11 16:02:17,868 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-2) [6288851b] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-11 16:02:17,910 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:17,917 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: d935e35
>2014-08-11 16:02:17,918 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] VDS rack is spm and moved from up calling resetIrs.
>2014-08-11 16:02:17,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 5ebbdd55
>2014-08-11 16:02:17,926 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4bfb1b8
>2014-08-11 16:02:17,932 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9
>2014-08-11 16:02:17,988 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, SpmStopVDSCommand, log id: 4bfb1b8
>2014-08-11 16:02:17,996 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, ResetIrsVDSCommand, log id: 5ebbdd55
>2014-08-11 16:02:18,002 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, SetVdsStatusVDSCommand, log id: d935e35
>2014-08-11 16:02:18,057 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL
>]
>2014-08-11 16:02:18,089 INFO  [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Running command: MaintenanceVdsCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:18,133 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [6288851b] Correlation ID: 6288851b, Job ID: c366ecce-d4db-4b00-9106-45a2f0998c3e, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin.
>2014-08-11 16:02:19,932 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack
>2014-08-11 16:02:19,961 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack.
>2014-08-11 16:02:19,962 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache
>2014-08-11 16:02:19,964 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache
>2014-08-11 16:02:19,969 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: cdfd24d
>2014-08-11 16:02:22,444 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, DisconnectStoragePoolVDSCommand, log id: cdfd24d
>2014-08-11 16:02:22,451 INFO  [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:02:22,471 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 26390da9
>2014-08-11 16:02:22,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] FINISH, DisconnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 26390da9
>2014-08-11 16:02:25,368 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 63fe3516-4170-402a-9dc8-6d3eab1c7933 status from Active to Unknown, reason : no reporting hosts
>2014-08-11 16:02:25,372 INFO  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts
>2014-08-11 16:02:25,378 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [13e2eaf6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:02:25,389 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [13e2eaf6] Correlation ID: 13e2eaf6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive.
>2014-08-11 16:02:30,570 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-11 16:02:30,571 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-11 16:02:30,573 INFO  [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [71c95186] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:30,614 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-11 16:02:30,615 WARN  [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic
>2014-08-11 16:02:30,622 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71c95186] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-11 16:02:30,639 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71c95186] Correlation ID: 71c95186, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin.
>2014-08-11 16:02:34,897 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [287267fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-11 16:02:34,931 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Running command: ActivateVdsCommand internal: false. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:34,934 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Before acquiring lock in order to prevent monitoring for host rack from data-center Default
>2014-08-11 16:02:34,935 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-41) [287267fa] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_INIT
>, sharedLocks= ]
>2014-08-11 16:02:34,948 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default
>2014-08-11 16:02:34,966 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 721eb9fe
>2014-08-11 16:02:34,973 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] FINISH, SetVdsStatusVDSCommand, log id: 721eb9fe
>2014-08-11 16:02:35,013 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default
>2014-08-11 16:02:35,021 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [287267fa] Correlation ID: 287267fa, Job ID: 0a945613-5f69-4fb9-8f1f-0fa218caf432, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin.
>2014-08-11 16:02:35,027 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS
>, sharedLocks= ]
>2014-08-11 16:02:38,055 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 1b79d63f
>2014-08-11 16:02:38,106 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 1b79d63f
>2014-08-11 16:02:38,134 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [33c8cfcd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:38,186 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [72966734] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:02:38,198 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:02:38,214 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 60859deb
>2014-08-11 16:02:43,607 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 60859deb
>2014-08-11 16:02:43,609 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] Host rack storage connection was succeeded 
>2014-08-11 16:02:43,618 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1c03780
>2014-08-11 16:02:45,119 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 1c03780
>2014-08-11 16:02:45,133 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70c0d85e
>2014-08-11 16:02:45,230 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 70c0d85e
>2014-08-11 16:02:45,239 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [229df7c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up.
>2014-08-11 16:02:45,245 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [229df7c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack.
>2014-08-11 16:02:45,268 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [4e9c0ea5] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS
>2014-08-11 16:02:45,275 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache
>2014-08-11 16:02:45,546 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default
>2014-08-11 16:02:45,552 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) starting spm on vds rack, storage pool Default, prevId -1, LVER -1
>2014-08-11 16:02:45,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71d49d48
>2014-08-11 16:02:45,603 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling started: taskId = c33e7f3a-9e54-4d1b-9431-a5eded1a66f6
>2014-08-11 16:03:07,746 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended: taskId = c33e7f3a-9e54-4d1b-9431-a5eded1a66f6 task status = finished
>2014-08-11 16:03:07,802 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended, spm status: SPM
>2014-08-11 16:03:07,805 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c33e7f3a-9e54-4d1b-9431-a5eded1a66f6), log id: 66f3dd52
>2014-08-11 16:03:07,849 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, HSMClearTaskVDSCommand, log id: 66f3dd52
>2014-08-11 16:03:07,854 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1094465d, log id: 71d49d48
>2014-08-11 16:03:07,862 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:07,868 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:07,907 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-42) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:03:17,960 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [1144b5b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:03:17,972 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [1144b5b] Correlation ID: 1144b5b, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:03:18,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [1144b5b] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:03:18,033 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [1144b5b] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:18,041 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [1144b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:18,042 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-22) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:03:28,095 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:03:28,161 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:03:28,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:28,180 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:28,183 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-91) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:03:38,300 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:03:38,433 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [67bcf03e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:03:38,436 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:38,443 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:38,444 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-34) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:03:48,500 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-50) [6d88073] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:03:48,565 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [6d88073] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:03:48,569 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [6d88073] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:48,575 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [6d88073] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:48,577 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-50) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:03:58,630 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [53a642c3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:03:58,696 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [53a642c3] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:03:58,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [53a642c3] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:03:58,705 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [53a642c3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:03:58,707 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:08,761 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [8bd4403] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:08,827 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [8bd4403] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:08,831 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [8bd4403] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:08,837 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [8bd4403] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:08,838 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-23) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:18,910 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [2249f537] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:18,936 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2249f537] Correlation ID: 2249f537, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:04:18,960 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [2249f537] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:18,966 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [2249f537] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:18,974 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2249f537] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:18,976 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-82) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:29,038 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [497a3d78] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:29,104 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [497a3d78] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:29,108 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [497a3d78] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:29,115 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [497a3d78] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:29,117 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-89) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:39,195 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [996c68a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:39,260 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [996c68a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:39,263 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [996c68a] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:39,270 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [996c68a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:39,272 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-54) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:49,329 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:49,394 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:49,398 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:49,406 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:49,407 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-45) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:04:59,463 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:04:59,567 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:04:59,572 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:04:59,591 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:04:59,595 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-19) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:05:09,705 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [1d849b94] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:05:09,789 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [1d849b94] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:05:09,792 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [1d849b94] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:05:09,796 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [1d849b94] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:05:09,797 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-35) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:05:19,884 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [59838f6f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:05:19,895 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [59838f6f] Correlation ID: 59838f6f, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:05:19,972 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [59838f6f] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:05:19,977 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [59838f6f] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:05:19,984 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [59838f6f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:05:19,985 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-76) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:05:30,043 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-12) [d351fc8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:05:30,109 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [d351fc8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:05:30,112 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [d351fc8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:05:30,118 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [d351fc8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:05:30,119 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-12) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:05:40,171 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [10593fed] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:05:40,236 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [10593fed] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:05:40,240 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [10593fed] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:05:40,247 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [10593fed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:05:40,249 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-100) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:05:50,300 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-29) [1ebede43] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:05:50,399 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) [1ebede43] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:05:50,403 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) [1ebede43] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:05:50,409 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [1ebede43] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:05:50,410 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-29) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:00,467 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:00,534 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:00,538 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:00,544 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:00,545 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-84) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:10,604 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [3dd51269] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:10,669 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [3dd51269] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:10,672 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [3dd51269] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:10,679 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [3dd51269] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:10,680 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-64) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:20,736 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [30a5360c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:20,747 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [30a5360c] Correlation ID: 30a5360c, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:06:20,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [30a5360c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:20,809 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [30a5360c] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:20,815 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [30a5360c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:20,816 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-6) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:30,873 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-81) [19324c69] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:30,939 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) [19324c69] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:30,942 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) [19324c69] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:30,965 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [19324c69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:30,966 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-81) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:41,019 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-37) [5147fe99] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:41,089 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [5147fe99] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:41,093 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [5147fe99] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:41,100 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5147fe99] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:41,102 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-37) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:06:51,171 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-73) [7ec672a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:06:51,242 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [7ec672a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:06:51,247 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [7ec672a] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:06:51,257 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7ec672a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:06:51,258 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-73) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:01,318 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-77) [2daf0726] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:01,385 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2daf0726] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:01,390 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2daf0726] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:01,397 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [2daf0726] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:01,399 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-77) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:11,586 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-57) [28141123] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:11,671 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [28141123] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:11,677 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [28141123] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:11,684 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [28141123] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:11,686 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-57) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:21,745 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [55e964ee] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:21,759 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [55e964ee] Correlation ID: 55e964ee, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:07:21,778 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [55e964ee] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:21,782 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [55e964ee] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:21,789 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [55e964ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:21,791 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-75) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:31,851 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-49) [64682f24] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:31,916 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [64682f24] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:31,919 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [64682f24] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:31,925 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [64682f24] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:31,927 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-49) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:41,982 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-92) [2707196a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:42,049 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) [2707196a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:42,052 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) [2707196a] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:42,058 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2707196a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:42,060 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-92) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:07:52,115 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [43def580] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:07:52,181 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [43def580] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:07:52,185 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [43def580] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:07:52,192 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [43def580] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:07:52,193 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-76) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:02,252 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [10064a3d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:02,317 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [10064a3d] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:02,320 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [10064a3d] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:02,327 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [10064a3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:02,328 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-24) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:12,415 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:12,484 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:12,488 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:12,495 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:12,496 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-13) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:22,565 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:22,579 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Correlation ID: 2fc9b3f7, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:08:22,702 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:22,706 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:22,713 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:22,716 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-78) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:32,778 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:32,805 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [306c8cb8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:32,809 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:32,816 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:32,818 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-7) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:42,871 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-83) [507a5afd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:42,937 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) [507a5afd] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:42,945 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) [507a5afd] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:42,954 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [507a5afd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:42,955 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-83) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:08:53,016 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:08:53,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [7352f1e1] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:08:53,087 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:08:53,097 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:08:53,099 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-89) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:02,450 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-11 16:09:02,468 WARN  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] CanDoAction of action ActivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_MASTER_STORAGE_DOMAIN_NOT_ACTIVE
>2014-08-11 16:09:02,471 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE
>, sharedLocks= ]
>2014-08-11 16:09:03,155 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:03,223 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:03,227 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:03,234 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:03,235 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-31) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:13,287 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:13,354 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [6e8cea09] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:13,358 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:13,365 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:13,366 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-66) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:14,858 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-12) [419a23bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-08-11 16:09:14,886 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage
>2014-08-11 16:09:14,892 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE
>, sharedLocks= ]
>2014-08-11 16:09:14,894 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/11/14 4:09 PM
>2014-08-11 16:09:14,899 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2014-08-11 16:09:14,904 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36910f4d
>2014-08-11 16:09:14,917 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 36910f4d
>2014-08-11 16:09:14,920 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 79339613
>2014-08-11 16:09:14,930 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-5) [306141b7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:15,305 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [306141b7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:15,310 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [306141b7] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:15,317 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [306141b7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:15,328 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [306141b7] Command [id=9a19a899-664b-429f-9fb4-1e8cb37e30ab]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, status=Unknown].
>2014-08-11 16:09:15,338 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [306141b7] Correlation ID: 419a23bc, Job ID: a236ea04-ec22-4bb0-ab95-4a33f15c1a7d, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain 35 (Data Center Default) by admin
>2014-08-11 16:09:23,419 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-18) [3e35b05] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:23,429 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3e35b05] Correlation ID: 3e35b05, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:09:23,487 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [3e35b05] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:23,491 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [3e35b05] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:23,498 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3e35b05] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:23,500 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-18) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:33,562 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:33,591 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2530e0a8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:33,594 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:33,600 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:33,602 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-67) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:43,656 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:43,722 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [37aaaaff] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:43,726 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:43,732 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:43,733 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-68) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:09:53,786 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-97) [3c896365] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:09:53,849 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [3c896365] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:09:53,853 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [3c896365] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:09:53,858 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [3c896365] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:09:53,860 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-97) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:03,916 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:03,947 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [2c1a1597] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:03,950 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:03,956 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:03,958 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-21) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:14,013 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [27d34e72] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:14,078 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [27d34e72] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:14,081 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [27d34e72] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:14,088 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [27d34e72] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:14,089 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:24,162 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:24,183 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Correlation ID: 3ea4a3b8, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:10:24,210 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:24,217 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:24,227 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:24,234 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-23) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:34,340 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:34,405 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [170dbdb3] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:34,411 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:34,418 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:34,420 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-62) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:44,511 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:44,581 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [3e89abaf] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:44,588 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:44,598 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:44,604 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-56) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:10:54,775 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:10:54,865 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [7c187b0c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:10:54,869 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:10:54,876 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:10:54,877 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-80) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:11:04,959 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [76e7601e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:11:05,030 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [76e7601e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:11:05,034 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [76e7601e] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:11:05,045 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [76e7601e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:11:05,046 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-45) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:11:15,099 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:11:15,169 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) [4b23bf35] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:11:15,177 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:11:15,184 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:11:15,185 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-43) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:11:25,254 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-96) [179dd399] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:11:25,265 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [179dd399] Correlation ID: 179dd399, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete.
>2014-08-11 16:11:25,283 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) [179dd399] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:11:25,287 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) [179dd399] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:11:25,293 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [179dd399] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:11:25,295 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-96) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>
>2014-08-11 16:11:35,348 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [76b61321] Running command: SetStoragePoolStatusCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool
>2014-08-11 16:11:35,382 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [76b61321] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default
>2014-08-11 16:11:35,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [76b61321] Initialize Irs proxy from vds: 10.34.62.208
>2014-08-11 16:11:35,394 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [76b61321] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208).
>2014-08-11 16:11:35,396 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-31) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException
>	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55]
>	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
>	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
>	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
>	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
>	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
>Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
>	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
>	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:]
>	at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:]
>	... 6 more
>

Actions: View

Attachments on bug 1128778: 925779